(APIServer pid=1215265) INFO 07-24 06:35:17 [api_utils.py:339] (APIServer pid=1215265) INFO 07-24 06:35:17 [api_utils.py:339] █ █ █▄ ▄█ (APIServer pid=1215265) INFO 07-24 06:35:17 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0 (APIServer pid=1215265) INFO 07-24 06:35:17 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/healed_dl/keep50_step0200 (APIServer pid=1215265) INFO 07-24 06:35:17 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1215265) INFO 07-24 06:35:17 [api_utils.py:339] (APIServer pid=1215265) INFO 07-24 06:35:17 [api_utils.py:273] non-default args: {'model_tag': 'outputs/healed_dl/keep50_step0200', 'default_chat_template_kwargs': {'enable_thinking': False}, 'host': '127.0.0.1', 'port': 8399, 'model': 'outputs/healed_dl/keep50_step0200', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['student'], 'pipeline_parallel_size': 2, 'gpu_memory_utilization': 0.9} (APIServer pid=1215265) INFO 07-24 06:35:17 [model.py:619] Resolved architecture: PrunedQwen3_5MoeForCausalLM (APIServer pid=1215265) INFO 07-24 06:35:17 [model.py:1776] Using max model len 8192 (APIServer pid=1215265) INFO 07-24 06:35:17 [vllm.py:1042] Asynchronous scheduling is enabled. (APIServer pid=1215265) WARNING 07-24 06:35:17 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (APIServer pid=1215265) WARNING 07-24 06:35:17 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (APIServer pid=1215265) INFO 07-24 06:35:17 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) (APIServer pid=1215265) INFO 07-24 06:35:18 [vllm.py:1322] Cudagraph is disabled under eager mode (APIServer pid=1215265) INFO 07-24 06:35:18 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant (EngineCore pid=1215449) INFO 07-24 06:35:29 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/healed_dl/keep50_step0200', speculative_config=None, tokenizer='outputs/healed_dl/keep50_step0200', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=2, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') (EngineCore pid=1215449) WARNING 07-24 06:35:29 [multiproc_executor.py:1067] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore pid=1215449) INFO 07-24 06:35:29 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.0.15 (local), world_size=2, local_world_size=2 (Worker pid=1215563) INFO 07-24 06:35:37 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:40045 backend=nccl (Worker pid=1215564) INFO 07-24 06:35:37 [parallel_state.py:1607] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:40045 backend=nccl (Worker pid=1215563) INFO 07-24 06:35:38 [pynccl.py:113] vLLM is using nccl==2.28.9 (Worker pid=1215563) INFO 07-24 06:35:39 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. (Worker pid=1215563) INFO 07-24 06:35:39 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (Worker pid=1215563) INFO 07-24 06:35:39 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. (Worker_PP0 pid=1215563) INFO 07-24 06:35:39 [gpu_model_runner.py:5209] Starting to load model outputs/healed_dl/keep50_step0200... (Worker_PP1 pid=1215564) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. (Worker_PP1 pid=1215564) warnings.warn('Grouped GEMM not available.') (Worker_PP0 pid=1215563) INFO 07-24 06:35:39 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). (Worker_PP0 pid=1215563) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. (Worker_PP0 pid=1215563) warnings.warn('Grouped GEMM not available.') (Worker_PP0 pid=1215563) INFO 07-24 06:35:40 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (Worker_PP0 pid=1215563) INFO 07-24 06:35:40 [flash_attn.py:718] Using FlashAttention version 2 (Worker_PP0 pid=1215563) INFO 07-24 06:35:40 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 34.55 GiB. Available RAM: 107.02 GiB. (Worker_PP0 pid=1215563) INFO 07-24 06:35:40 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. (Worker_PP0 pid=1215563) Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00= mamba page size. (Worker_PP0 pid=1215563) INFO 07-24 06:36:33 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. (Worker_PP1 pid=1215564) INFO 07-24 06:36:42 [interface.py:890] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. (Worker_PP1 pid=1215564) INFO 07-24 06:36:42 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. (Worker_PP0 pid=1215563) INFO 07-24 06:36:44 [gpu_worker.py:538] Available KV cache memory: 3.56 GiB (EngineCore pid=1215449) INFO 07-24 06:36:44 [kv_cache_utils.py:2146] GPU KV cache size: 157,372 tokens (EngineCore pid=1215449) INFO 07-24 06:36:44 [kv_cache_utils.py:2147] Maximum concurrency for 8,192 tokens per request: 19.21x (Worker_PP1 pid=1215564) INFO 07-24 06:36:45 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. (Worker_PP0 pid=1215563) INFO 07-24 06:36:45 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. (Worker_PP0 pid=1215563) INFO 07-24 06:36:45 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (Worker_PP1 pid=1215564) INFO 07-24 06:36:45 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (EngineCore pid=1215449) INFO 07-24 06:36:45 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.52 s (EngineCore pid=1215449) INFO 07-24 06:36:48 [vllm.py:1042] Asynchronous scheduling is enabled. (EngineCore pid=1215449) WARNING 07-24 06:36:48 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (EngineCore pid=1215449) WARNING 07-24 06:36:48 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (EngineCore pid=1215449) INFO 07-24 06:36:48 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) (EngineCore pid=1215449) INFO 07-24 06:36:48 [vllm.py:1322] Cudagraph is disabled under eager mode (EngineCore pid=1215449) INFO 07-24 06:36:48 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant (APIServer pid=1215265) INFO 07-24 06:36:48 [api_server.py:612] Supported tasks: ['generate'] (APIServer pid=1215265) INFO 07-24 06:36:52 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. (APIServer pid=1215265) INFO 07-24 06:36:52 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8399 (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:37] Available routes are: (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /docs, Methods: HEAD, GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /redoc, Methods: HEAD, GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /load, Methods: GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /version, Methods: GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /health, Methods: GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /metrics, Methods: GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /tokenize, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /detokenize, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/models, Methods: GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /ping, Methods: GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /ping, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /invocations, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/chat/completions, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/responses, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/completions, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/messages, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /generative_scoring, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/completions/render, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /v1/completions/derender, Methods: POST (APIServer pid=1215265) INFO 07-24 06:36:52 [launcher.py:46] Route: /inference/v1/generate, Methods: POST (APIServer pid=1215265) INFO: Started server process [1215265] (APIServer pid=1215265) INFO: Waiting for application startup. (APIServer pid=1215265) INFO: Application startup complete. (APIServer pid=1215265) INFO: 127.0.0.1:44998 - "GET /health HTTP/1.1" 200 OK (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _zero_kv_blocks_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _fused_post_conv_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _build_route_rows. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) /home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/distributed/parallel_state.py:813: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_new.cpp:1586.) (Worker_PP0 pid=1215563) object_tensor = torch.frombuffer(pickle.dumps(obj), dtype=torch.uint8) [rank0]:[W724 06:37:27.285850302 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) [rank1]:[W724 06:37:27.286233953 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_recurrent_gated_delta_rule_packed_decode_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1215563) WARNING 07-24 06:37:27 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_sigmoid_gating_delta_rule_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (APIServer pid=1215265) INFO 07-24 06:37:32 [loggers.py:273] Engine 000: Avg prompt throughput: 3056.9 tokens/s, Avg generation throughput: 17.0 tokens/s, Running: 23 reqs, Waiting: 41 reqs, GPU KV cache usage: 37.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 06:37:42 [loggers.py:273] Engine 000: Avg prompt throughput: 6088.3 tokens/s, Avg generation throughput: 147.6 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:37:52 [loggers.py:273] Engine 000: Avg prompt throughput: 923.2 tokens/s, Avg generation throughput: 460.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:38:02 [loggers.py:273] Engine 000: Avg prompt throughput: 2775.9 tokens/s, Avg generation throughput: 351.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:38:12 [loggers.py:273] Engine 000: Avg prompt throughput: 1357.3 tokens/s, Avg generation throughput: 408.9 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:38:22 [loggers.py:273] Engine 000: Avg prompt throughput: 1904.8 tokens/s, Avg generation throughput: 404.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:38:32 [loggers.py:273] Engine 000: Avg prompt throughput: 471.9 tokens/s, Avg generation throughput: 493.6 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:38:42 [loggers.py:273] Engine 000: Avg prompt throughput: 818.0 tokens/s, Avg generation throughput: 448.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:38:52 [loggers.py:273] Engine 000: Avg prompt throughput: 786.9 tokens/s, Avg generation throughput: 442.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:39:02 [loggers.py:273] Engine 000: Avg prompt throughput: 1247.4 tokens/s, Avg generation throughput: 415.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:39:12 [loggers.py:273] Engine 000: Avg prompt throughput: 1824.5 tokens/s, Avg generation throughput: 357.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:39:22 [loggers.py:273] Engine 000: Avg prompt throughput: 1684.7 tokens/s, Avg generation throughput: 388.7 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:39:32 [loggers.py:273] Engine 000: Avg prompt throughput: 1139.8 tokens/s, Avg generation throughput: 412.2 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:39:42 [loggers.py:273] Engine 000: Avg prompt throughput: 1451.4 tokens/s, Avg generation throughput: 381.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:39:52 [loggers.py:273] Engine 000: Avg prompt throughput: 1440.0 tokens/s, Avg generation throughput: 395.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:40:02 [loggers.py:273] Engine 000: Avg prompt throughput: 805.9 tokens/s, Avg generation throughput: 397.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:40:12 [loggers.py:273] Engine 000: Avg prompt throughput: 1682.3 tokens/s, Avg generation throughput: 376.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:40:22 [loggers.py:273] Engine 000: Avg prompt throughput: 2983.8 tokens/s, Avg generation throughput: 318.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:40:32 [loggers.py:273] Engine 000: Avg prompt throughput: 966.1 tokens/s, Avg generation throughput: 394.2 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:40:42 [loggers.py:273] Engine 000: Avg prompt throughput: 2441.3 tokens/s, Avg generation throughput: 337.9 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:40:52 [loggers.py:273] Engine 000: Avg prompt throughput: 1913.8 tokens/s, Avg generation throughput: 365.0 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:41:02 [loggers.py:273] Engine 000: Avg prompt throughput: 1556.7 tokens/s, Avg generation throughput: 406.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:41:12 [loggers.py:273] Engine 000: Avg prompt throughput: 2763.2 tokens/s, Avg generation throughput: 334.0 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:41:22 [loggers.py:273] Engine 000: Avg prompt throughput: 1808.5 tokens/s, Avg generation throughput: 378.6 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:41:32 [loggers.py:273] Engine 000: Avg prompt throughput: 1449.8 tokens/s, Avg generation throughput: 387.1 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:41:42 [loggers.py:273] Engine 000: Avg prompt throughput: 2067.7 tokens/s, Avg generation throughput: 372.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:41:52 [loggers.py:273] Engine 000: Avg prompt throughput: 1410.5 tokens/s, Avg generation throughput: 404.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:42:02 [loggers.py:273] Engine 000: Avg prompt throughput: 1698.6 tokens/s, Avg generation throughput: 395.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:42:12 [loggers.py:273] Engine 000: Avg prompt throughput: 1191.2 tokens/s, Avg generation throughput: 411.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:42:22 [loggers.py:273] Engine 000: Avg prompt throughput: 1599.8 tokens/s, Avg generation throughput: 377.0 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:42:32 [loggers.py:273] Engine 000: Avg prompt throughput: 1136.6 tokens/s, Avg generation throughput: 424.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:42:42 [loggers.py:273] Engine 000: Avg prompt throughput: 2027.2 tokens/s, Avg generation throughput: 358.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:42:52 [loggers.py:273] Engine 000: Avg prompt throughput: 1429.7 tokens/s, Avg generation throughput: 406.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:43:02 [loggers.py:273] Engine 000: Avg prompt throughput: 1952.0 tokens/s, Avg generation throughput: 354.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:43:12 [loggers.py:273] Engine 000: Avg prompt throughput: 650.4 tokens/s, Avg generation throughput: 446.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:43:22 [loggers.py:273] Engine 000: Avg prompt throughput: 1623.4 tokens/s, Avg generation throughput: 383.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:43:32 [loggers.py:273] Engine 000: Avg prompt throughput: 2463.0 tokens/s, Avg generation throughput: 342.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:43:42 [loggers.py:273] Engine 000: Avg prompt throughput: 1382.9 tokens/s, Avg generation throughput: 411.7 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:43:52 [loggers.py:273] Engine 000: Avg prompt throughput: 1313.7 tokens/s, Avg generation throughput: 411.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:44:02 [loggers.py:273] Engine 000: Avg prompt throughput: 1690.8 tokens/s, Avg generation throughput: 357.8 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:44:12 [loggers.py:273] Engine 000: Avg prompt throughput: 1468.9 tokens/s, Avg generation throughput: 410.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:44:22 [loggers.py:273] Engine 000: Avg prompt throughput: 1596.5 tokens/s, Avg generation throughput: 388.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:44:32 [loggers.py:273] Engine 000: Avg prompt throughput: 1092.1 tokens/s, Avg generation throughput: 399.9 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:44:42 [loggers.py:273] Engine 000: Avg prompt throughput: 2076.0 tokens/s, Avg generation throughput: 372.3 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:44:52 [loggers.py:273] Engine 000: Avg prompt throughput: 1159.8 tokens/s, Avg generation throughput: 423.5 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:45:02 [loggers.py:273] Engine 000: Avg prompt throughput: 1758.9 tokens/s, Avg generation throughput: 379.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:45:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1299.2 tokens/s, Avg generation throughput: 410.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:45:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1110.8 tokens/s, Avg generation throughput: 415.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:45:33 [loggers.py:273] Engine 000: Avg prompt throughput: 639.5 tokens/s, Avg generation throughput: 448.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:45:43 [loggers.py:273] Engine 000: Avg prompt throughput: 782.4 tokens/s, Avg generation throughput: 435.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:45:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1495.4 tokens/s, Avg generation throughput: 382.6 tokens/s, Running: 49 reqs, Waiting: 13 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:46:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1748.1 tokens/s, Avg generation throughput: 393.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:46:13 [loggers.py:273] Engine 000: Avg prompt throughput: 486.7 tokens/s, Avg generation throughput: 424.2 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:46:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2065.7 tokens/s, Avg generation throughput: 354.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:46:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1288.7 tokens/s, Avg generation throughput: 384.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:46:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1274.4 tokens/s, Avg generation throughput: 403.5 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:46:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1589.2 tokens/s, Avg generation throughput: 381.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:47:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1349.9 tokens/s, Avg generation throughput: 391.9 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:47:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1320.9 tokens/s, Avg generation throughput: 405.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:47:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1178.8 tokens/s, Avg generation throughput: 381.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:47:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2058.2 tokens/s, Avg generation throughput: 352.2 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:47:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1452.7 tokens/s, Avg generation throughput: 380.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:47:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2104.2 tokens/s, Avg generation throughput: 374.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:48:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1872.7 tokens/s, Avg generation throughput: 367.8 tokens/s, Running: 52 reqs, Waiting: 10 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:48:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1881.4 tokens/s, Avg generation throughput: 394.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:48:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1144.5 tokens/s, Avg generation throughput: 395.1 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:48:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1631.0 tokens/s, Avg generation throughput: 380.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:48:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1547.6 tokens/s, Avg generation throughput: 385.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:48:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1584.9 tokens/s, Avg generation throughput: 377.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:49:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1145.2 tokens/s, Avg generation throughput: 389.7 tokens/s, Running: 48 reqs, Waiting: 14 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:49:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1928.2 tokens/s, Avg generation throughput: 383.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:49:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1392.0 tokens/s, Avg generation throughput: 385.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:49:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2232.5 tokens/s, Avg generation throughput: 361.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:49:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1907.0 tokens/s, Avg generation throughput: 394.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:49:53 [loggers.py:273] Engine 000: Avg prompt throughput: 952.2 tokens/s, Avg generation throughput: 463.4 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:50:03 [loggers.py:273] Engine 000: Avg prompt throughput: 361.3 tokens/s, Avg generation throughput: 510.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:50:13 [loggers.py:273] Engine 000: Avg prompt throughput: 694.2 tokens/s, Avg generation throughput: 475.7 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:50:23 [loggers.py:273] Engine 000: Avg prompt throughput: 896.5 tokens/s, Avg generation throughput: 463.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:50:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1137.9 tokens/s, Avg generation throughput: 438.0 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:50:43 [loggers.py:273] Engine 000: Avg prompt throughput: 705.0 tokens/s, Avg generation throughput: 488.2 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:50:53 [loggers.py:273] Engine 000: Avg prompt throughput: 928.9 tokens/s, Avg generation throughput: 484.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:51:03 [loggers.py:273] Engine 000: Avg prompt throughput: 278.6 tokens/s, Avg generation throughput: 527.8 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:51:13 [loggers.py:273] Engine 000: Avg prompt throughput: 614.7 tokens/s, Avg generation throughput: 491.0 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:51:23 [loggers.py:273] Engine 000: Avg prompt throughput: 460.6 tokens/s, Avg generation throughput: 503.3 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:51:33 [loggers.py:273] Engine 000: Avg prompt throughput: 458.7 tokens/s, Avg generation throughput: 497.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:51:43 [loggers.py:273] Engine 000: Avg prompt throughput: 243.1 tokens/s, Avg generation throughput: 506.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:51:53 [loggers.py:273] Engine 000: Avg prompt throughput: 480.2 tokens/s, Avg generation throughput: 486.5 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:52:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1277.3 tokens/s, Avg generation throughput: 443.3 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:52:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1202.9 tokens/s, Avg generation throughput: 442.0 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:52:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1071.9 tokens/s, Avg generation throughput: 461.2 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:52:33 [loggers.py:273] Engine 000: Avg prompt throughput: 121.2 tokens/s, Avg generation throughput: 525.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:52:43 [loggers.py:273] Engine 000: Avg prompt throughput: 844.2 tokens/s, Avg generation throughput: 464.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:52:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1415.2 tokens/s, Avg generation throughput: 456.8 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:53:03 [loggers.py:273] Engine 000: Avg prompt throughput: 477.4 tokens/s, Avg generation throughput: 533.0 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:53:13 [loggers.py:273] Engine 000: Avg prompt throughput: 467.5 tokens/s, Avg generation throughput: 510.6 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:53:23 [loggers.py:273] Engine 000: Avg prompt throughput: 384.4 tokens/s, Avg generation throughput: 523.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:53:33 [loggers.py:273] Engine 000: Avg prompt throughput: 689.2 tokens/s, Avg generation throughput: 500.5 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:53:43 [loggers.py:273] Engine 000: Avg prompt throughput: 750.0 tokens/s, Avg generation throughput: 490.0 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:53:53 [loggers.py:273] Engine 000: Avg prompt throughput: 354.8 tokens/s, Avg generation throughput: 514.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:54:03 [loggers.py:273] Engine 000: Avg prompt throughput: 666.6 tokens/s, Avg generation throughput: 470.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:54:13 [loggers.py:273] Engine 000: Avg prompt throughput: 218.4 tokens/s, Avg generation throughput: 486.2 tokens/s, Running: 50 reqs, Waiting: 11 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:54:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1541.0 tokens/s, Avg generation throughput: 421.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:54:33 [loggers.py:273] Engine 000: Avg prompt throughput: 857.3 tokens/s, Avg generation throughput: 457.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:54:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1879.1 tokens/s, Avg generation throughput: 427.9 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:54:53 [loggers.py:273] Engine 000: Avg prompt throughput: 576.0 tokens/s, Avg generation throughput: 506.0 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:55:03 [loggers.py:273] Engine 000: Avg prompt throughput: 997.7 tokens/s, Avg generation throughput: 470.9 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:55:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1433.8 tokens/s, Avg generation throughput: 443.8 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:55:23 [loggers.py:273] Engine 000: Avg prompt throughput: 805.5 tokens/s, Avg generation throughput: 487.6 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:55:33 [loggers.py:273] Engine 000: Avg prompt throughput: 545.6 tokens/s, Avg generation throughput: 509.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:55:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1073.3 tokens/s, Avg generation throughput: 482.6 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:55:53 [loggers.py:273] Engine 000: Avg prompt throughput: 970.5 tokens/s, Avg generation throughput: 498.5 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:56:03 [loggers.py:273] Engine 000: Avg prompt throughput: 567.8 tokens/s, Avg generation throughput: 503.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:56:13 [loggers.py:273] Engine 000: Avg prompt throughput: 567.5 tokens/s, Avg generation throughput: 501.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:56:23 [loggers.py:273] Engine 000: Avg prompt throughput: 231.0 tokens/s, Avg generation throughput: 520.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:56:33 [loggers.py:273] Engine 000: Avg prompt throughput: 244.0 tokens/s, Avg generation throughput: 476.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:56:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1094.2 tokens/s, Avg generation throughput: 435.6 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:56:53 [loggers.py:273] Engine 000: Avg prompt throughput: 835.3 tokens/s, Avg generation throughput: 485.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:57:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1394.4 tokens/s, Avg generation throughput: 453.4 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:57:13 [loggers.py:273] Engine 000: Avg prompt throughput: 604.2 tokens/s, Avg generation throughput: 516.2 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:57:23 [loggers.py:273] Engine 000: Avg prompt throughput: 901.4 tokens/s, Avg generation throughput: 486.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:57:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1424.9 tokens/s, Avg generation throughput: 460.8 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:57:43 [loggers.py:273] Engine 000: Avg prompt throughput: 364.0 tokens/s, Avg generation throughput: 539.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:57:53 [loggers.py:273] Engine 000: Avg prompt throughput: 691.3 tokens/s, Avg generation throughput: 506.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:58:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1112.3 tokens/s, Avg generation throughput: 464.5 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:58:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1168.7 tokens/s, Avg generation throughput: 470.8 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:58:23 [loggers.py:273] Engine 000: Avg prompt throughput: 266.6 tokens/s, Avg generation throughput: 522.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:58:33 [loggers.py:273] Engine 000: Avg prompt throughput: 698.5 tokens/s, Avg generation throughput: 493.3 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:58:43 [loggers.py:273] Engine 000: Avg prompt throughput: 591.4 tokens/s, Avg generation throughput: 501.0 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:58:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 520.8 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:59:03 [loggers.py:273] Engine 000: Avg prompt throughput: 240.7 tokens/s, Avg generation throughput: 496.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:59:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1072.1 tokens/s, Avg generation throughput: 453.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:59:23 [loggers.py:273] Engine 000: Avg prompt throughput: 689.5 tokens/s, Avg generation throughput: 470.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:59:33 [loggers.py:273] Engine 000: Avg prompt throughput: 834.3 tokens/s, Avg generation throughput: 475.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:59:43 [loggers.py:273] Engine 000: Avg prompt throughput: 891.7 tokens/s, Avg generation throughput: 475.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 06:59:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1829.6 tokens/s, Avg generation throughput: 425.5 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:00:03 [loggers.py:273] Engine 000: Avg prompt throughput: 458.3 tokens/s, Avg generation throughput: 517.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:00:13 [loggers.py:273] Engine 000: Avg prompt throughput: 619.8 tokens/s, Avg generation throughput: 503.3 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:00:23 [loggers.py:273] Engine 000: Avg prompt throughput: 335.7 tokens/s, Avg generation throughput: 511.6 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:00:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1207.8 tokens/s, Avg generation throughput: 473.8 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:00:43 [loggers.py:273] Engine 000: Avg prompt throughput: 234.2 tokens/s, Avg generation throughput: 520.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:00:53 [loggers.py:273] Engine 000: Avg prompt throughput: 699.0 tokens/s, Avg generation throughput: 512.4 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:01:03 [loggers.py:273] Engine 000: Avg prompt throughput: 343.4 tokens/s, Avg generation throughput: 516.3 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:01:13 [loggers.py:273] Engine 000: Avg prompt throughput: 229.6 tokens/s, Avg generation throughput: 494.6 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:01:23 [loggers.py:273] Engine 000: Avg prompt throughput: 715.3 tokens/s, Avg generation throughput: 487.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:01:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1199.4 tokens/s, Avg generation throughput: 444.9 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:01:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1142.2 tokens/s, Avg generation throughput: 446.3 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:01:53 [loggers.py:273] Engine 000: Avg prompt throughput: 724.3 tokens/s, Avg generation throughput: 501.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:02:03 [loggers.py:273] Engine 000: Avg prompt throughput: 597.3 tokens/s, Avg generation throughput: 506.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:02:13 [loggers.py:273] Engine 000: Avg prompt throughput: 691.6 tokens/s, Avg generation throughput: 491.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:02:23 [loggers.py:273] Engine 000: Avg prompt throughput: 823.4 tokens/s, Avg generation throughput: 496.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:02:33 [loggers.py:273] Engine 000: Avg prompt throughput: 585.7 tokens/s, Avg generation throughput: 520.3 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:02:43 [loggers.py:273] Engine 000: Avg prompt throughput: 114.5 tokens/s, Avg generation throughput: 527.1 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:02:53 [loggers.py:273] Engine 000: Avg prompt throughput: 932.7 tokens/s, Avg generation throughput: 483.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:03:03 [loggers.py:273] Engine 000: Avg prompt throughput: 779.7 tokens/s, Avg generation throughput: 473.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:03:13 [loggers.py:273] Engine 000: Avg prompt throughput: 877.9 tokens/s, Avg generation throughput: 456.9 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:03:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1273.9 tokens/s, Avg generation throughput: 427.5 tokens/s, Running: 52 reqs, Waiting: 9 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:03:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1073.2 tokens/s, Avg generation throughput: 472.0 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:03:43 [loggers.py:273] Engine 000: Avg prompt throughput: 810.5 tokens/s, Avg generation throughput: 476.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:03:53 [loggers.py:273] Engine 000: Avg prompt throughput: 841.0 tokens/s, Avg generation throughput: 492.3 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:04:03 [loggers.py:273] Engine 000: Avg prompt throughput: 714.3 tokens/s, Avg generation throughput: 500.8 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:04:13 [loggers.py:273] Engine 000: Avg prompt throughput: 816.2 tokens/s, Avg generation throughput: 486.2 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:04:23 [loggers.py:273] Engine 000: Avg prompt throughput: 324.7 tokens/s, Avg generation throughput: 515.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:04:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1110.8 tokens/s, Avg generation throughput: 457.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:04:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1088.1 tokens/s, Avg generation throughput: 469.0 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:04:53 [loggers.py:273] Engine 000: Avg prompt throughput: 696.8 tokens/s, Avg generation throughput: 499.0 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:05:03 [loggers.py:273] Engine 000: Avg prompt throughput: 241.7 tokens/s, Avg generation throughput: 539.8 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:05:13 [loggers.py:273] Engine 000: Avg prompt throughput: 465.8 tokens/s, Avg generation throughput: 503.2 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:05:23 [loggers.py:273] Engine 000: Avg prompt throughput: 114.2 tokens/s, Avg generation throughput: 522.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:05:33 [loggers.py:273] Engine 000: Avg prompt throughput: 237.4 tokens/s, Avg generation throughput: 489.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:05:43 [loggers.py:273] Engine 000: Avg prompt throughput: 738.6 tokens/s, Avg generation throughput: 435.6 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:05:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1095.2 tokens/s, Avg generation throughput: 456.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:06:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1150.7 tokens/s, Avg generation throughput: 470.3 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:06:13 [loggers.py:273] Engine 000: Avg prompt throughput: 597.3 tokens/s, Avg generation throughput: 508.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:06:23 [loggers.py:273] Engine 000: Avg prompt throughput: 342.0 tokens/s, Avg generation throughput: 538.7 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:06:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1003.9 tokens/s, Avg generation throughput: 446.5 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:06:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1096.8 tokens/s, Avg generation throughput: 455.3 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:06:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1311.0 tokens/s, Avg generation throughput: 449.7 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:07:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1315.4 tokens/s, Avg generation throughput: 457.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:07:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1067.6 tokens/s, Avg generation throughput: 486.4 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:07:23 [loggers.py:273] Engine 000: Avg prompt throughput: 606.0 tokens/s, Avg generation throughput: 531.4 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:07:33 [loggers.py:273] Engine 000: Avg prompt throughput: 382.0 tokens/s, Avg generation throughput: 530.2 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:07:43 [loggers.py:273] Engine 000: Avg prompt throughput: 811.1 tokens/s, Avg generation throughput: 497.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:07:53 [loggers.py:273] Engine 000: Avg prompt throughput: 580.2 tokens/s, Avg generation throughput: 506.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:08:03 [loggers.py:273] Engine 000: Avg prompt throughput: 676.2 tokens/s, Avg generation throughput: 479.6 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:08:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1370.2 tokens/s, Avg generation throughput: 439.2 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:08:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1405.3 tokens/s, Avg generation throughput: 436.5 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:08:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1110.8 tokens/s, Avg generation throughput: 485.8 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:08:43 [loggers.py:273] Engine 000: Avg prompt throughput: 528.5 tokens/s, Avg generation throughput: 502.7 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:08:53 [loggers.py:273] Engine 000: Avg prompt throughput: 607.5 tokens/s, Avg generation throughput: 505.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:09:03 [loggers.py:273] Engine 000: Avg prompt throughput: 681.0 tokens/s, Avg generation throughput: 477.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:09:13 [loggers.py:273] Engine 000: Avg prompt throughput: 818.0 tokens/s, Avg generation throughput: 441.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:09:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1166.6 tokens/s, Avg generation throughput: 432.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:09:33 [loggers.py:273] Engine 000: Avg prompt throughput: 688.8 tokens/s, Avg generation throughput: 460.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:09:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1226.5 tokens/s, Avg generation throughput: 418.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:09:53 [loggers.py:273] Engine 000: Avg prompt throughput: 540.0 tokens/s, Avg generation throughput: 426.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:10:03 [loggers.py:273] Engine 000: Avg prompt throughput: 179.5 tokens/s, Avg generation throughput: 491.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:10:13 [loggers.py:273] Engine 000: Avg prompt throughput: 547.9 tokens/s, Avg generation throughput: 443.7 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:10:23 [loggers.py:273] Engine 000: Avg prompt throughput: 504.5 tokens/s, Avg generation throughput: 441.7 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:10:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 451.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:10:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1511.3 tokens/s, Avg generation throughput: 370.7 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:10:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1104.6 tokens/s, Avg generation throughput: 408.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:11:03 [loggers.py:273] Engine 000: Avg prompt throughput: 863.0 tokens/s, Avg generation throughput: 427.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:11:13 [loggers.py:273] Engine 000: Avg prompt throughput: 502.9 tokens/s, Avg generation throughput: 446.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:11:23 [loggers.py:273] Engine 000: Avg prompt throughput: 330.4 tokens/s, Avg generation throughput: 455.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:11:33 [loggers.py:273] Engine 000: Avg prompt throughput: 509.1 tokens/s, Avg generation throughput: 433.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:11:43 [loggers.py:273] Engine 000: Avg prompt throughput: 338.1 tokens/s, Avg generation throughput: 422.6 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:11:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1027.1 tokens/s, Avg generation throughput: 405.6 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:12:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1404.5 tokens/s, Avg generation throughput: 390.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:12:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1070.8 tokens/s, Avg generation throughput: 403.4 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:12:23 [loggers.py:273] Engine 000: Avg prompt throughput: 526.3 tokens/s, Avg generation throughput: 449.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:12:33 [loggers.py:273] Engine 000: Avg prompt throughput: 335.0 tokens/s, Avg generation throughput: 440.1 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:12:43 [loggers.py:273] Engine 000: Avg prompt throughput: 172.6 tokens/s, Avg generation throughput: 443.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:12:53 [loggers.py:273] Engine 000: Avg prompt throughput: 515.7 tokens/s, Avg generation throughput: 406.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:13:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1124.2 tokens/s, Avg generation throughput: 392.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:13:13 [loggers.py:273] Engine 000: Avg prompt throughput: 338.5 tokens/s, Avg generation throughput: 430.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:13:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1359.1 tokens/s, Avg generation throughput: 370.7 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:13:33 [loggers.py:273] Engine 000: Avg prompt throughput: 349.3 tokens/s, Avg generation throughput: 442.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:13:43 [loggers.py:273] Engine 000: Avg prompt throughput: 826.9 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:13:53 [loggers.py:273] Engine 000: Avg prompt throughput: 557.4 tokens/s, Avg generation throughput: 427.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:14:03 [loggers.py:273] Engine 000: Avg prompt throughput: 530.8 tokens/s, Avg generation throughput: 414.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:14:13 [loggers.py:273] Engine 000: Avg prompt throughput: 695.3 tokens/s, Avg generation throughput: 395.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:14:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1201.3 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:14:33 [loggers.py:273] Engine 000: Avg prompt throughput: 862.1 tokens/s, Avg generation throughput: 393.0 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:14:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1053.5 tokens/s, Avg generation throughput: 410.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:14:53 [loggers.py:273] Engine 000: Avg prompt throughput: 336.2 tokens/s, Avg generation throughput: 439.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:15:03 [loggers.py:273] Engine 000: Avg prompt throughput: 164.9 tokens/s, Avg generation throughput: 469.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:15:13 [loggers.py:273] Engine 000: Avg prompt throughput: 348.3 tokens/s, Avg generation throughput: 437.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:15:23 [loggers.py:273] Engine 000: Avg prompt throughput: 845.9 tokens/s, Avg generation throughput: 407.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:15:33 [loggers.py:273] Engine 000: Avg prompt throughput: 162.7 tokens/s, Avg generation throughput: 445.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:15:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1213.6 tokens/s, Avg generation throughput: 374.1 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:15:53 [loggers.py:273] Engine 000: Avg prompt throughput: 560.4 tokens/s, Avg generation throughput: 436.5 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:16:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1386.6 tokens/s, Avg generation throughput: 392.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:16:13 [loggers.py:273] Engine 000: Avg prompt throughput: 513.8 tokens/s, Avg generation throughput: 418.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:16:23 [loggers.py:273] Engine 000: Avg prompt throughput: 869.8 tokens/s, Avg generation throughput: 383.1 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:16:33 [loggers.py:273] Engine 000: Avg prompt throughput: 648.3 tokens/s, Avg generation throughput: 435.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:16:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1162.8 tokens/s, Avg generation throughput: 408.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:16:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1030.2 tokens/s, Avg generation throughput: 413.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:17:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1732.4 tokens/s, Avg generation throughput: 380.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:17:13 [loggers.py:273] Engine 000: Avg prompt throughput: 852.2 tokens/s, Avg generation throughput: 436.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:17:23 [loggers.py:273] Engine 000: Avg prompt throughput: 171.9 tokens/s, Avg generation throughput: 479.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:17:33 [loggers.py:273] Engine 000: Avg prompt throughput: 324.8 tokens/s, Avg generation throughput: 458.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:17:43 [loggers.py:273] Engine 000: Avg prompt throughput: 162.1 tokens/s, Avg generation throughput: 439.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 07:17:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 477.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:18:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1184.6 tokens/s, Avg generation throughput: 372.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:18:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1025.9 tokens/s, Avg generation throughput: 404.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:18:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1036.0 tokens/s, Avg generation throughput: 404.9 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:18:33 [loggers.py:273] Engine 000: Avg prompt throughput: 506.9 tokens/s, Avg generation throughput: 451.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:18:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1349.3 tokens/s, Avg generation throughput: 398.6 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:18:53 [loggers.py:273] Engine 000: Avg prompt throughput: 515.9 tokens/s, Avg generation throughput: 451.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:19:03 [loggers.py:273] Engine 000: Avg prompt throughput: 330.0 tokens/s, Avg generation throughput: 436.3 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:19:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1728.7 tokens/s, Avg generation throughput: 367.2 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:19:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1904.0 tokens/s, Avg generation throughput: 387.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:19:33 [loggers.py:273] Engine 000: Avg prompt throughput: 664.6 tokens/s, Avg generation throughput: 445.4 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:19:43 [loggers.py:273] Engine 000: Avg prompt throughput: 822.5 tokens/s, Avg generation throughput: 447.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:19:53 [loggers.py:273] Engine 000: Avg prompt throughput: 718.1 tokens/s, Avg generation throughput: 445.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:20:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 462.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 07:20:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 474.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:20:23 [loggers.py:273] Engine 000: Avg prompt throughput: 516.9 tokens/s, Avg generation throughput: 397.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:20:33 [loggers.py:273] Engine 000: Avg prompt throughput: 908.2 tokens/s, Avg generation throughput: 418.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:20:43 [loggers.py:273] Engine 000: Avg prompt throughput: 331.9 tokens/s, Avg generation throughput: 436.9 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:20:53 [loggers.py:273] Engine 000: Avg prompt throughput: 166.3 tokens/s, Avg generation throughput: 444.3 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:21:03 [loggers.py:273] Engine 000: Avg prompt throughput: 881.6 tokens/s, Avg generation throughput: 405.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:21:13 [loggers.py:273] Engine 000: Avg prompt throughput: 165.2 tokens/s, Avg generation throughput: 425.9 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:21:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1198.0 tokens/s, Avg generation throughput: 386.4 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:21:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1905.4 tokens/s, Avg generation throughput: 340.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:21:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2223.6 tokens/s, Avg generation throughput: 367.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:21:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 483.0 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:22:03 [loggers.py:273] Engine 000: Avg prompt throughput: 688.9 tokens/s, Avg generation throughput: 450.0 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:22:13 [loggers.py:273] Engine 000: Avg prompt throughput: 873.4 tokens/s, Avg generation throughput: 423.4 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:22:23 [loggers.py:273] Engine 000: Avg prompt throughput: 173.0 tokens/s, Avg generation throughput: 467.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:22:33 [loggers.py:273] Engine 000: Avg prompt throughput: 164.3 tokens/s, Avg generation throughput: 453.4 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:22:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1910.7 tokens/s, Avg generation throughput: 367.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:22:53 [loggers.py:273] Engine 000: Avg prompt throughput: 860.0 tokens/s, Avg generation throughput: 427.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:23:03 [loggers.py:273] Engine 000: Avg prompt throughput: 704.8 tokens/s, Avg generation throughput: 441.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:23:13 [loggers.py:273] Engine 000: Avg prompt throughput: 170.5 tokens/s, Avg generation throughput: 453.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:23:23 [loggers.py:273] Engine 000: Avg prompt throughput: 517.1 tokens/s, Avg generation throughput: 425.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:23:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 444.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:23:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1409.4 tokens/s, Avg generation throughput: 358.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:23:53 [loggers.py:273] Engine 000: Avg prompt throughput: 720.2 tokens/s, Avg generation throughput: 403.2 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:24:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1373.7 tokens/s, Avg generation throughput: 388.0 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:24:13 [loggers.py:273] Engine 000: Avg prompt throughput: 369.3 tokens/s, Avg generation throughput: 450.9 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:24:23 [loggers.py:273] Engine 000: Avg prompt throughput: 351.6 tokens/s, Avg generation throughput: 460.4 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:24:33 [loggers.py:273] Engine 000: Avg prompt throughput: 524.6 tokens/s, Avg generation throughput: 446.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:24:43 [loggers.py:273] Engine 000: Avg prompt throughput: 840.8 tokens/s, Avg generation throughput: 413.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:24:53 [loggers.py:273] Engine 000: Avg prompt throughput: 341.0 tokens/s, Avg generation throughput: 431.9 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:25:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1884.8 tokens/s, Avg generation throughput: 371.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:25:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1047.0 tokens/s, Avg generation throughput: 406.5 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:25:23 [loggers.py:273] Engine 000: Avg prompt throughput: 692.4 tokens/s, Avg generation throughput: 450.7 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:25:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1069.7 tokens/s, Avg generation throughput: 421.7 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:25:43 [loggers.py:273] Engine 000: Avg prompt throughput: 492.2 tokens/s, Avg generation throughput: 445.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 07:25:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 479.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:26:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1183.9 tokens/s, Avg generation throughput: 391.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:26:13 [loggers.py:273] Engine 000: Avg prompt throughput: 905.1 tokens/s, Avg generation throughput: 404.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:26:23 [loggers.py:273] Engine 000: Avg prompt throughput: 972.5 tokens/s, Avg generation throughput: 412.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:26:33 [loggers.py:273] Engine 000: Avg prompt throughput: 658.0 tokens/s, Avg generation throughput: 441.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:26:43 [loggers.py:273] Engine 000: Avg prompt throughput: 173.3 tokens/s, Avg generation throughput: 444.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:26:53 [loggers.py:273] Engine 000: Avg prompt throughput: 561.1 tokens/s, Avg generation throughput: 427.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:27:03 [loggers.py:273] Engine 000: Avg prompt throughput: 522.9 tokens/s, Avg generation throughput: 422.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:27:13 [loggers.py:273] Engine 000: Avg prompt throughput: 170.8 tokens/s, Avg generation throughput: 429.5 tokens/s, Running: 44 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:27:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1745.4 tokens/s, Avg generation throughput: 368.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:27:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1175.1 tokens/s, Avg generation throughput: 378.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:27:43 [loggers.py:273] Engine 000: Avg prompt throughput: 723.8 tokens/s, Avg generation throughput: 441.4 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:27:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1218.1 tokens/s, Avg generation throughput: 415.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 07:28:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:28:13 [loggers.py:273] Engine 000: Avg prompt throughput: 322.6 tokens/s, Avg generation throughput: 448.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:28:23 [loggers.py:273] Engine 000: Avg prompt throughput: 489.5 tokens/s, Avg generation throughput: 428.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:28:33 [loggers.py:273] Engine 000: Avg prompt throughput: 853.0 tokens/s, Avg generation throughput: 390.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:28:43 [loggers.py:273] Engine 000: Avg prompt throughput: 512.3 tokens/s, Avg generation throughput: 414.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:28:53 [loggers.py:273] Engine 000: Avg prompt throughput: 879.4 tokens/s, Avg generation throughput: 422.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:29:03 [loggers.py:273] Engine 000: Avg prompt throughput: 662.2 tokens/s, Avg generation throughput: 401.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:29:13 [loggers.py:273] Engine 000: Avg prompt throughput: 697.1 tokens/s, Avg generation throughput: 422.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:29:23 [loggers.py:273] Engine 000: Avg prompt throughput: 523.8 tokens/s, Avg generation throughput: 430.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:29:33 [loggers.py:273] Engine 000: Avg prompt throughput: 173.4 tokens/s, Avg generation throughput: 439.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:29:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1784.7 tokens/s, Avg generation throughput: 354.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:29:53 [loggers.py:273] Engine 000: Avg prompt throughput: 669.3 tokens/s, Avg generation throughput: 424.0 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:30:03 [loggers.py:273] Engine 000: Avg prompt throughput: 828.9 tokens/s, Avg generation throughput: 439.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:30:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1332.0 tokens/s, Avg generation throughput: 398.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:30:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 463.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:30:33 [loggers.py:273] Engine 000: Avg prompt throughput: 495.5 tokens/s, Avg generation throughput: 435.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:30:43 [loggers.py:273] Engine 000: Avg prompt throughput: 363.4 tokens/s, Avg generation throughput: 436.7 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:30:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1206.0 tokens/s, Avg generation throughput: 402.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:31:03 [loggers.py:273] Engine 000: Avg prompt throughput: 734.4 tokens/s, Avg generation throughput: 403.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:31:13 [loggers.py:273] Engine 000: Avg prompt throughput: 530.1 tokens/s, Avg generation throughput: 432.0 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:31:23 [loggers.py:273] Engine 000: Avg prompt throughput: 897.6 tokens/s, Avg generation throughput: 417.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:31:33 [loggers.py:273] Engine 000: Avg prompt throughput: 492.0 tokens/s, Avg generation throughput: 431.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:31:43 [loggers.py:273] Engine 000: Avg prompt throughput: 692.5 tokens/s, Avg generation throughput: 408.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:31:53 [loggers.py:273] Engine 000: Avg prompt throughput: 201.4 tokens/s, Avg generation throughput: 455.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:32:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1876.7 tokens/s, Avg generation throughput: 338.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:32:13 [loggers.py:273] Engine 000: Avg prompt throughput: 663.1 tokens/s, Avg generation throughput: 432.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:32:23 [loggers.py:273] Engine 000: Avg prompt throughput: 329.1 tokens/s, Avg generation throughput: 446.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:32:33 [loggers.py:273] Engine 000: Avg prompt throughput: 335.8 tokens/s, Avg generation throughput: 465.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:32:43 [loggers.py:273] Engine 000: Avg prompt throughput: 178.1 tokens/s, Avg generation throughput: 452.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:32:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 458.0 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:33:03 [loggers.py:273] Engine 000: Avg prompt throughput: 695.1 tokens/s, Avg generation throughput: 405.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:33:13 [loggers.py:273] Engine 000: Avg prompt throughput: 886.3 tokens/s, Avg generation throughput: 413.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:33:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1195.1 tokens/s, Avg generation throughput: 387.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:33:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1050.8 tokens/s, Avg generation throughput: 396.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:33:43 [loggers.py:273] Engine 000: Avg prompt throughput: 667.4 tokens/s, Avg generation throughput: 441.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:33:53 [loggers.py:273] Engine 000: Avg prompt throughput: 491.0 tokens/s, Avg generation throughput: 430.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:34:03 [loggers.py:273] Engine 000: Avg prompt throughput: 864.0 tokens/s, Avg generation throughput: 412.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:34:13 [loggers.py:273] Engine 000: Avg prompt throughput: 535.3 tokens/s, Avg generation throughput: 427.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:34:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2282.2 tokens/s, Avg generation throughput: 341.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:34:33 [loggers.py:273] Engine 000: Avg prompt throughput: 554.8 tokens/s, Avg generation throughput: 443.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:34:43 [loggers.py:273] Engine 000: Avg prompt throughput: 703.5 tokens/s, Avg generation throughput: 431.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:34:53 [loggers.py:273] Engine 000: Avg prompt throughput: 503.8 tokens/s, Avg generation throughput: 455.7 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:35:03 [loggers.py:273] Engine 000: Avg prompt throughput: 175.3 tokens/s, Avg generation throughput: 458.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:35:13 [loggers.py:273] Engine 000: Avg prompt throughput: 187.7 tokens/s, Avg generation throughput: 450.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:35:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 440.5 tokens/s, Running: 43 reqs, Waiting: 20 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:35:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1019.0 tokens/s, Avg generation throughput: 389.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:35:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1378.0 tokens/s, Avg generation throughput: 378.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:35:53 [loggers.py:273] Engine 000: Avg prompt throughput: 674.9 tokens/s, Avg generation throughput: 415.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:36:03 [loggers.py:273] Engine 000: Avg prompt throughput: 173.2 tokens/s, Avg generation throughput: 450.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:36:13 [loggers.py:273] Engine 000: Avg prompt throughput: 550.3 tokens/s, Avg generation throughput: 430.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:36:23 [loggers.py:273] Engine 000: Avg prompt throughput: 698.2 tokens/s, Avg generation throughput: 409.8 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:36:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1028.6 tokens/s, Avg generation throughput: 382.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:36:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2171.2 tokens/s, Avg generation throughput: 352.1 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:36:53 [loggers.py:273] Engine 000: Avg prompt throughput: 652.4 tokens/s, Avg generation throughput: 448.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:37:03 [loggers.py:273] Engine 000: Avg prompt throughput: 483.2 tokens/s, Avg generation throughput: 455.7 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:37:13 [loggers.py:273] Engine 000: Avg prompt throughput: 495.1 tokens/s, Avg generation throughput: 450.3 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 07:37:23 [loggers.py:273] Engine 000: Avg prompt throughput: 158.8 tokens/s, Avg generation throughput: 471.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:37:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 437.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:37:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1033.5 tokens/s, Avg generation throughput: 399.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:37:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1060.5 tokens/s, Avg generation throughput: 406.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:38:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1114.3 tokens/s, Avg generation throughput: 404.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:38:13 [loggers.py:273] Engine 000: Avg prompt throughput: 372.7 tokens/s, Avg generation throughput: 432.0 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:38:23 [loggers.py:273] Engine 000: Avg prompt throughput: 362.1 tokens/s, Avg generation throughput: 458.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:38:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.7 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:38:43 [loggers.py:273] Engine 000: Avg prompt throughput: 671.6 tokens/s, Avg generation throughput: 381.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:38:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1257.9 tokens/s, Avg generation throughput: 377.3 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:39:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2193.6 tokens/s, Avg generation throughput: 350.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:39:13 [loggers.py:273] Engine 000: Avg prompt throughput: 870.1 tokens/s, Avg generation throughput: 425.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:39:23 [loggers.py:273] Engine 000: Avg prompt throughput: 368.6 tokens/s, Avg generation throughput: 470.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:39:33 [loggers.py:273] Engine 000: Avg prompt throughput: 176.1 tokens/s, Avg generation throughput: 482.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:39:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 452.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:39:53 [loggers.py:273] Engine 000: Avg prompt throughput: 158.4 tokens/s, Avg generation throughput: 431.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:40:03 [loggers.py:273] Engine 000: Avg prompt throughput: 995.8 tokens/s, Avg generation throughput: 390.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:40:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1207.1 tokens/s, Avg generation throughput: 387.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:40:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1014.8 tokens/s, Avg generation throughput: 418.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:40:33 [loggers.py:273] Engine 000: Avg prompt throughput: 534.0 tokens/s, Avg generation throughput: 446.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:40:43 [loggers.py:273] Engine 000: Avg prompt throughput: 318.8 tokens/s, Avg generation throughput: 445.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:40:53 [loggers.py:273] Engine 000: Avg prompt throughput: 526.5 tokens/s, Avg generation throughput: 435.8 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:41:03 [loggers.py:273] Engine 000: Avg prompt throughput: 679.8 tokens/s, Avg generation throughput: 400.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:41:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1502.1 tokens/s, Avg generation throughput: 351.9 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:41:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2478.3 tokens/s, Avg generation throughput: 333.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:41:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1046.2 tokens/s, Avg generation throughput: 435.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:41:43 [loggers.py:273] Engine 000: Avg prompt throughput: 330.8 tokens/s, Avg generation throughput: 470.1 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:41:53 [loggers.py:273] Engine 000: Avg prompt throughput: 493.6 tokens/s, Avg generation throughput: 470.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:42:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 462.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:42:13 [loggers.py:273] Engine 000: Avg prompt throughput: 545.2 tokens/s, Avg generation throughput: 416.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:42:23 [loggers.py:273] Engine 000: Avg prompt throughput: 688.7 tokens/s, Avg generation throughput: 422.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:42:33 [loggers.py:273] Engine 000: Avg prompt throughput: 868.1 tokens/s, Avg generation throughput: 414.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:42:43 [loggers.py:273] Engine 000: Avg prompt throughput: 871.4 tokens/s, Avg generation throughput: 424.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:42:53 [loggers.py:273] Engine 000: Avg prompt throughput: 356.5 tokens/s, Avg generation throughput: 444.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:43:03 [loggers.py:273] Engine 000: Avg prompt throughput: 380.7 tokens/s, Avg generation throughput: 438.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:43:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 443.7 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:43:23 [loggers.py:273] Engine 000: Avg prompt throughput: 535.4 tokens/s, Avg generation throughput: 398.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:43:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1550.5 tokens/s, Avg generation throughput: 366.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:43:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1927.3 tokens/s, Avg generation throughput: 362.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:43:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1402.2 tokens/s, Avg generation throughput: 402.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:44:03 [loggers.py:273] Engine 000: Avg prompt throughput: 352.0 tokens/s, Avg generation throughput: 457.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:44:13 [loggers.py:273] Engine 000: Avg prompt throughput: 851.8 tokens/s, Avg generation throughput: 436.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:44:23 [loggers.py:273] Engine 000: Avg prompt throughput: 496.6 tokens/s, Avg generation throughput: 442.1 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:44:33 [loggers.py:273] Engine 000: Avg prompt throughput: 517.1 tokens/s, Avg generation throughput: 445.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:44:43 [loggers.py:273] Engine 000: Avg prompt throughput: 159.5 tokens/s, Avg generation throughput: 418.6 tokens/s, Running: 44 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:44:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1973.7 tokens/s, Avg generation throughput: 364.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:45:03 [loggers.py:273] Engine 000: Avg prompt throughput: 868.7 tokens/s, Avg generation throughput: 411.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:45:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 454.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:45:23 [loggers.py:273] Engine 000: Avg prompt throughput: 347.3 tokens/s, Avg generation throughput: 436.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:45:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:45:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 427.6 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:45:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1534.5 tokens/s, Avg generation throughput: 369.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:46:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1646.4 tokens/s, Avg generation throughput: 363.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:46:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1028.6 tokens/s, Avg generation throughput: 404.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:46:23 [loggers.py:273] Engine 000: Avg prompt throughput: 336.9 tokens/s, Avg generation throughput: 440.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:46:33 [loggers.py:273] Engine 000: Avg prompt throughput: 685.2 tokens/s, Avg generation throughput: 428.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:46:43 [loggers.py:273] Engine 000: Avg prompt throughput: 702.7 tokens/s, Avg generation throughput: 424.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:46:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 462.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:47:03 [loggers.py:273] Engine 000: Avg prompt throughput: 568.1 tokens/s, Avg generation throughput: 424.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:47:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1909.2 tokens/s, Avg generation throughput: 346.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:47:23 [loggers.py:273] Engine 000: Avg prompt throughput: 678.5 tokens/s, Avg generation throughput: 419.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:47:33 [loggers.py:273] Engine 000: Avg prompt throughput: 348.8 tokens/s, Avg generation throughput: 426.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:47:43 [loggers.py:273] Engine 000: Avg prompt throughput: 178.3 tokens/s, Avg generation throughput: 447.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:47:53 [loggers.py:273] Engine 000: Avg prompt throughput: 868.0 tokens/s, Avg generation throughput: 413.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:48:03 [loggers.py:273] Engine 000: Avg prompt throughput: 518.8 tokens/s, Avg generation throughput: 431.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:48:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1563.8 tokens/s, Avg generation throughput: 364.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:48:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1536.7 tokens/s, Avg generation throughput: 374.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:48:33 [loggers.py:273] Engine 000: Avg prompt throughput: 862.3 tokens/s, Avg generation throughput: 418.1 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:48:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1061.6 tokens/s, Avg generation throughput: 431.5 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:48:53 [loggers.py:273] Engine 000: Avg prompt throughput: 164.8 tokens/s, Avg generation throughput: 476.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:49:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1050.5 tokens/s, Avg generation throughput: 408.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:49:13 [loggers.py:273] Engine 000: Avg prompt throughput: 556.1 tokens/s, Avg generation throughput: 436.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:49:23 [loggers.py:273] Engine 000: Avg prompt throughput: 526.7 tokens/s, Avg generation throughput: 415.7 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:49:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1060.6 tokens/s, Avg generation throughput: 404.5 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:49:43 [loggers.py:273] Engine 000: Avg prompt throughput: 505.8 tokens/s, Avg generation throughput: 458.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:49:53 [loggers.py:273] Engine 000: Avg prompt throughput: 335.8 tokens/s, Avg generation throughput: 429.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:50:03 [loggers.py:273] Engine 000: Avg prompt throughput: 341.7 tokens/s, Avg generation throughput: 442.3 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:50:13 [loggers.py:273] Engine 000: Avg prompt throughput: 680.9 tokens/s, Avg generation throughput: 400.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:50:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1202.5 tokens/s, Avg generation throughput: 381.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:50:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1014.1 tokens/s, Avg generation throughput: 395.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:50:43 [loggers.py:273] Engine 000: Avg prompt throughput: 824.7 tokens/s, Avg generation throughput: 406.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:50:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1151.3 tokens/s, Avg generation throughput: 402.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:51:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1016.7 tokens/s, Avg generation throughput: 410.5 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:51:13 [loggers.py:273] Engine 000: Avg prompt throughput: 737.2 tokens/s, Avg generation throughput: 441.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:51:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1514.6 tokens/s, Avg generation throughput: 361.5 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:51:33 [loggers.py:273] Engine 000: Avg prompt throughput: 173.0 tokens/s, Avg generation throughput: 470.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:51:43 [loggers.py:273] Engine 000: Avg prompt throughput: 705.4 tokens/s, Avg generation throughput: 436.4 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:51:53 [loggers.py:273] Engine 000: Avg prompt throughput: 498.9 tokens/s, Avg generation throughput: 440.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:52:03 [loggers.py:273] Engine 000: Avg prompt throughput: 369.3 tokens/s, Avg generation throughput: 455.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:52:13 [loggers.py:273] Engine 000: Avg prompt throughput: 873.2 tokens/s, Avg generation throughput: 422.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:52:23 [loggers.py:273] Engine 000: Avg prompt throughput: 678.8 tokens/s, Avg generation throughput: 418.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:52:33 [loggers.py:273] Engine 000: Avg prompt throughput: 171.8 tokens/s, Avg generation throughput: 440.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:52:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1462.4 tokens/s, Avg generation throughput: 364.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:52:53 [loggers.py:273] Engine 000: Avg prompt throughput: 874.0 tokens/s, Avg generation throughput: 405.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:53:03 [loggers.py:273] Engine 000: Avg prompt throughput: 874.5 tokens/s, Avg generation throughput: 405.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:53:13 [loggers.py:273] Engine 000: Avg prompt throughput: 820.7 tokens/s, Avg generation throughput: 416.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:53:23 [loggers.py:273] Engine 000: Avg prompt throughput: 652.9 tokens/s, Avg generation throughput: 408.1 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:53:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1111.2 tokens/s, Avg generation throughput: 372.8 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:53:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1207.2 tokens/s, Avg generation throughput: 389.6 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:53:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1650.0 tokens/s, Avg generation throughput: 361.4 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:54:03 [loggers.py:273] Engine 000: Avg prompt throughput: 477.7 tokens/s, Avg generation throughput: 414.4 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:54:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1548.4 tokens/s, Avg generation throughput: 366.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:54:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1513.4 tokens/s, Avg generation throughput: 384.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:54:33 [loggers.py:273] Engine 000: Avg prompt throughput: 827.4 tokens/s, Avg generation throughput: 417.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:54:43 [loggers.py:273] Engine 000: Avg prompt throughput: 975.8 tokens/s, Avg generation throughput: 391.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:54:53 [loggers.py:273] Engine 000: Avg prompt throughput: 494.4 tokens/s, Avg generation throughput: 410.1 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:55:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1595.4 tokens/s, Avg generation throughput: 368.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:55:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2236.2 tokens/s, Avg generation throughput: 334.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:55:23 [loggers.py:273] Engine 000: Avg prompt throughput: 836.0 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:55:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1446.0 tokens/s, Avg generation throughput: 399.0 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:55:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1365.3 tokens/s, Avg generation throughput: 396.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:55:53 [loggers.py:273] Engine 000: Avg prompt throughput: 728.6 tokens/s, Avg generation throughput: 427.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:56:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1185.6 tokens/s, Avg generation throughput: 397.6 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:56:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1467.6 tokens/s, Avg generation throughput: 397.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:56:23 [loggers.py:273] Engine 000: Avg prompt throughput: 647.4 tokens/s, Avg generation throughput: 441.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:56:33 [loggers.py:273] Engine 000: Avg prompt throughput: 801.9 tokens/s, Avg generation throughput: 421.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:56:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1234.1 tokens/s, Avg generation throughput: 385.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:56:53 [loggers.py:273] Engine 000: Avg prompt throughput: 967.3 tokens/s, Avg generation throughput: 410.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:57:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1136.8 tokens/s, Avg generation throughput: 402.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:57:13 [loggers.py:273] Engine 000: Avg prompt throughput: 641.2 tokens/s, Avg generation throughput: 421.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:57:23 [loggers.py:273] Engine 000: Avg prompt throughput: 326.7 tokens/s, Avg generation throughput: 434.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:57:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1108.7 tokens/s, Avg generation throughput: 379.1 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:57:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1684.1 tokens/s, Avg generation throughput: 374.0 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:57:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1648.9 tokens/s, Avg generation throughput: 375.7 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:58:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1489.4 tokens/s, Avg generation throughput: 393.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:58:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1302.6 tokens/s, Avg generation throughput: 393.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:58:23 [loggers.py:273] Engine 000: Avg prompt throughput: 685.0 tokens/s, Avg generation throughput: 427.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:58:33 [loggers.py:273] Engine 000: Avg prompt throughput: 684.0 tokens/s, Avg generation throughput: 430.0 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:58:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1468.2 tokens/s, Avg generation throughput: 387.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:58:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1317.7 tokens/s, Avg generation throughput: 393.7 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:59:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1281.1 tokens/s, Avg generation throughput: 394.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:59:13 [loggers.py:273] Engine 000: Avg prompt throughput: 663.9 tokens/s, Avg generation throughput: 417.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:59:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1047.0 tokens/s, Avg generation throughput: 407.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:59:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1028.6 tokens/s, Avg generation throughput: 392.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:59:43 [loggers.py:273] Engine 000: Avg prompt throughput: 832.7 tokens/s, Avg generation throughput: 399.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 07:59:53 [loggers.py:273] Engine 000: Avg prompt throughput: 658.0 tokens/s, Avg generation throughput: 422.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:00:03 [loggers.py:273] Engine 000: Avg prompt throughput: 484.7 tokens/s, Avg generation throughput: 433.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:00:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1139.9 tokens/s, Avg generation throughput: 362.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:00:23 [loggers.py:273] Engine 000: Avg prompt throughput: 489.5 tokens/s, Avg generation throughput: 412.5 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:00:33 [loggers.py:273] Engine 000: Avg prompt throughput: 834.6 tokens/s, Avg generation throughput: 402.4 tokens/s, Running: 46 reqs, Waiting: 16 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:00:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1502.7 tokens/s, Avg generation throughput: 378.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:00:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1020.2 tokens/s, Avg generation throughput: 398.9 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:01:03 [loggers.py:273] Engine 000: Avg prompt throughput: 163.5 tokens/s, Avg generation throughput: 451.0 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:01:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1011.9 tokens/s, Avg generation throughput: 413.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:01:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1685.9 tokens/s, Avg generation throughput: 378.7 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:01:33 [loggers.py:273] Engine 000: Avg prompt throughput: 853.4 tokens/s, Avg generation throughput: 408.6 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:01:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1123.7 tokens/s, Avg generation throughput: 403.5 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:01:53 [loggers.py:273] Engine 000: Avg prompt throughput: 963.5 tokens/s, Avg generation throughput: 416.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:02:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1235.5 tokens/s, Avg generation throughput: 392.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:02:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1517.4 tokens/s, Avg generation throughput: 375.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:02:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1143.7 tokens/s, Avg generation throughput: 400.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:02:33 [loggers.py:273] Engine 000: Avg prompt throughput: 320.2 tokens/s, Avg generation throughput: 448.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:02:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1126.0 tokens/s, Avg generation throughput: 401.4 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:02:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1138.1 tokens/s, Avg generation throughput: 380.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:03:03 [loggers.py:273] Engine 000: Avg prompt throughput: 810.1 tokens/s, Avg generation throughput: 406.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:03:13 [loggers.py:273] Engine 000: Avg prompt throughput: 480.9 tokens/s, Avg generation throughput: 431.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:03:23 [loggers.py:273] Engine 000: Avg prompt throughput: 853.8 tokens/s, Avg generation throughput: 417.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:03:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1322.7 tokens/s, Avg generation throughput: 389.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:03:43 [loggers.py:273] Engine 000: Avg prompt throughput: 471.1 tokens/s, Avg generation throughput: 421.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:03:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1126.1 tokens/s, Avg generation throughput: 371.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:04:03 [loggers.py:273] Engine 000: Avg prompt throughput: 889.9 tokens/s, Avg generation throughput: 401.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:04:13 [loggers.py:273] Engine 000: Avg prompt throughput: 962.7 tokens/s, Avg generation throughput: 409.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:04:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1066.6 tokens/s, Avg generation throughput: 390.3 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:04:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2391.0 tokens/s, Avg generation throughput: 344.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:04:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1912.6 tokens/s, Avg generation throughput: 375.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:04:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2171.9 tokens/s, Avg generation throughput: 398.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:05:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1741.0 tokens/s, Avg generation throughput: 400.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:05:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1614.0 tokens/s, Avg generation throughput: 417.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:05:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1466.2 tokens/s, Avg generation throughput: 421.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:05:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2416.9 tokens/s, Avg generation throughput: 374.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:05:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1781.6 tokens/s, Avg generation throughput: 397.8 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:05:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1853.3 tokens/s, Avg generation throughput: 406.7 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:06:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1448.5 tokens/s, Avg generation throughput: 425.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:06:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1336.9 tokens/s, Avg generation throughput: 435.7 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:06:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1642.5 tokens/s, Avg generation throughput: 425.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:06:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1996.3 tokens/s, Avg generation throughput: 396.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:06:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1472.6 tokens/s, Avg generation throughput: 415.0 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:06:53 [loggers.py:273] Engine 000: Avg prompt throughput: 3049.5 tokens/s, Avg generation throughput: 344.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:07:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1848.4 tokens/s, Avg generation throughput: 416.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:07:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1932.5 tokens/s, Avg generation throughput: 390.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:07:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1517.4 tokens/s, Avg generation throughput: 449.8 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:07:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1309.7 tokens/s, Avg generation throughput: 437.2 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:07:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1701.0 tokens/s, Avg generation throughput: 425.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:07:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1984.1 tokens/s, Avg generation throughput: 398.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:08:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1924.0 tokens/s, Avg generation throughput: 408.8 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:08:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1586.0 tokens/s, Avg generation throughput: 455.0 tokens/s, Running: 54 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:08:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2662.6 tokens/s, Avg generation throughput: 371.9 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:08:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1615.6 tokens/s, Avg generation throughput: 444.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:08:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1394.0 tokens/s, Avg generation throughput: 431.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:08:53 [loggers.py:273] Engine 000: Avg prompt throughput: 961.9 tokens/s, Avg generation throughput: 453.6 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:09:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1622.4 tokens/s, Avg generation throughput: 421.0 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:09:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1727.3 tokens/s, Avg generation throughput: 425.3 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:09:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1866.6 tokens/s, Avg generation throughput: 419.8 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:09:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1928.9 tokens/s, Avg generation throughput: 410.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:09:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1536.1 tokens/s, Avg generation throughput: 431.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:09:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1219.2 tokens/s, Avg generation throughput: 434.8 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:10:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2653.5 tokens/s, Avg generation throughput: 361.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:10:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1726.1 tokens/s, Avg generation throughput: 425.2 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:10:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1738.3 tokens/s, Avg generation throughput: 407.2 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:10:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1653.7 tokens/s, Avg generation throughput: 429.2 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:10:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1092.1 tokens/s, Avg generation throughput: 449.8 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:10:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1351.0 tokens/s, Avg generation throughput: 439.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:11:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1878.7 tokens/s, Avg generation throughput: 409.7 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:11:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1363.3 tokens/s, Avg generation throughput: 413.6 tokens/s, Running: 52 reqs, Waiting: 10 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:11:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1803.8 tokens/s, Avg generation throughput: 412.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:11:33 [loggers.py:273] Engine 000: Avg prompt throughput: 939.2 tokens/s, Avg generation throughput: 460.0 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:11:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1513.9 tokens/s, Avg generation throughput: 428.5 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:11:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1833.9 tokens/s, Avg generation throughput: 388.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:12:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1202.3 tokens/s, Avg generation throughput: 436.4 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:12:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1891.0 tokens/s, Avg generation throughput: 407.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:12:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1797.1 tokens/s, Avg generation throughput: 402.1 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:12:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1319.6 tokens/s, Avg generation throughput: 445.8 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:12:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2081.6 tokens/s, Avg generation throughput: 408.8 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:12:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2022.3 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:13:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1910.4 tokens/s, Avg generation throughput: 405.3 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:13:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1501.3 tokens/s, Avg generation throughput: 457.9 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:13:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1820.5 tokens/s, Avg generation throughput: 423.4 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:13:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1707.4 tokens/s, Avg generation throughput: 405.7 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:13:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1743.7 tokens/s, Avg generation throughput: 427.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:13:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1747.3 tokens/s, Avg generation throughput: 424.7 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:14:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1367.9 tokens/s, Avg generation throughput: 425.7 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:14:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1392.5 tokens/s, Avg generation throughput: 416.2 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:14:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1391.2 tokens/s, Avg generation throughput: 441.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:14:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1321.2 tokens/s, Avg generation throughput: 449.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:14:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1485.5 tokens/s, Avg generation throughput: 428.1 tokens/s, Running: 51 reqs, Waiting: 10 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:14:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1059.5 tokens/s, Avg generation throughput: 457.1 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:15:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2065.2 tokens/s, Avg generation throughput: 392.9 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:15:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1115.7 tokens/s, Avg generation throughput: 441.1 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:15:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1374.9 tokens/s, Avg generation throughput: 417.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:15:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1286.3 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:15:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1874.7 tokens/s, Avg generation throughput: 380.3 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:15:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1067.8 tokens/s, Avg generation throughput: 421.8 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:16:03 [loggers.py:273] Engine 000: Avg prompt throughput: 726.8 tokens/s, Avg generation throughput: 492.3 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:16:13 [loggers.py:273] Engine 000: Avg prompt throughput: 583.3 tokens/s, Avg generation throughput: 456.7 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:16:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1287.5 tokens/s, Avg generation throughput: 416.1 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:16:33 [loggers.py:273] Engine 000: Avg prompt throughput: 951.4 tokens/s, Avg generation throughput: 451.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:16:43 [loggers.py:273] Engine 000: Avg prompt throughput: 133.0 tokens/s, Avg generation throughput: 503.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:16:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:17:03 [loggers.py:273] Engine 000: Avg prompt throughput: 293.1 tokens/s, Avg generation throughput: 472.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:17:13 [loggers.py:273] Engine 000: Avg prompt throughput: 425.9 tokens/s, Avg generation throughput: 453.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:17:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 472.2 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:17:33 [loggers.py:273] Engine 000: Avg prompt throughput: 567.4 tokens/s, Avg generation throughput: 462.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:17:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 479.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:17:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 478.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:18:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1638.8 tokens/s, Avg generation throughput: 364.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:18:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1424.0 tokens/s, Avg generation throughput: 413.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:18:23 [loggers.py:273] Engine 000: Avg prompt throughput: 691.0 tokens/s, Avg generation throughput: 449.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:18:33 [loggers.py:273] Engine 000: Avg prompt throughput: 486.9 tokens/s, Avg generation throughput: 464.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:18:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1697.2 tokens/s, Avg generation throughput: 393.4 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:18:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1382.3 tokens/s, Avg generation throughput: 448.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:19:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 515.8 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:19:13 [loggers.py:273] Engine 000: Avg prompt throughput: 141.3 tokens/s, Avg generation throughput: 509.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:19:23 [loggers.py:273] Engine 000: Avg prompt throughput: 140.7 tokens/s, Avg generation throughput: 480.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:19:33 [loggers.py:273] Engine 000: Avg prompt throughput: 320.9 tokens/s, Avg generation throughput: 485.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:19:43 [loggers.py:273] Engine 000: Avg prompt throughput: 150.0 tokens/s, Avg generation throughput: 449.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:19:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 490.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:20:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 465.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:20:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 453.7 tokens/s, Running: 44 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:20:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1395.3 tokens/s, Avg generation throughput: 371.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:20:33 [loggers.py:273] Engine 000: Avg prompt throughput: 821.4 tokens/s, Avg generation throughput: 421.4 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:20:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1174.2 tokens/s, Avg generation throughput: 417.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:20:53 [loggers.py:273] Engine 000: Avg prompt throughput: 615.5 tokens/s, Avg generation throughput: 458.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:21:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1980.5 tokens/s, Avg generation throughput: 371.9 tokens/s, Running: 51 reqs, Waiting: 11 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:21:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1828.1 tokens/s, Avg generation throughput: 418.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:21:23 [loggers.py:273] Engine 000: Avg prompt throughput: 163.2 tokens/s, Avg generation throughput: 515.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:21:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 506.7 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:21:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 515.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:21:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 491.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:22:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 473.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:22:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 455.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:22:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 472.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:22:33 [loggers.py:273] Engine 000: Avg prompt throughput: 135.3 tokens/s, Avg generation throughput: 436.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:22:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1181.3 tokens/s, Avg generation throughput: 397.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:22:53 [loggers.py:273] Engine 000: Avg prompt throughput: 991.5 tokens/s, Avg generation throughput: 408.6 tokens/s, Running: 47 reqs, Waiting: 15 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:23:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1299.5 tokens/s, Avg generation throughput: 420.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:23:13 [loggers.py:273] Engine 000: Avg prompt throughput: 480.4 tokens/s, Avg generation throughput: 464.5 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:23:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2371.5 tokens/s, Avg generation throughput: 339.3 tokens/s, Running: 52 reqs, Waiting: 10 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:23:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1339.1 tokens/s, Avg generation throughput: 453.2 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:23:43 [loggers.py:273] Engine 000: Avg prompt throughput: 137.8 tokens/s, Avg generation throughput: 513.6 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:23:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 517.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:24:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 494.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:24:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 486.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:24:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 486.3 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:24:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 446.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:24:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 451.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:24:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 430.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:25:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1124.6 tokens/s, Avg generation throughput: 368.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:25:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1365.3 tokens/s, Avg generation throughput: 390.6 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:25:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1503.1 tokens/s, Avg generation throughput: 417.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:25:33 [loggers.py:273] Engine 000: Avg prompt throughput: 275.1 tokens/s, Avg generation throughput: 469.3 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:25:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2518.1 tokens/s, Avg generation throughput: 332.5 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:25:53 [loggers.py:273] Engine 000: Avg prompt throughput: 704.4 tokens/s, Avg generation throughput: 512.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:26:03 [loggers.py:273] Engine 000: Avg prompt throughput: 455.5 tokens/s, Avg generation throughput: 510.7 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:26:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 531.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:26:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 461.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:26:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 508.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:26:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 483.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:26:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 474.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:27:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 463.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:27:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 445.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:27:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1048.3 tokens/s, Avg generation throughput: 372.2 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:27:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1734.8 tokens/s, Avg generation throughput: 367.5 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:27:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1779.4 tokens/s, Avg generation throughput: 400.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:27:53 [loggers.py:273] Engine 000: Avg prompt throughput: 591.4 tokens/s, Avg generation throughput: 473.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:28:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2551.5 tokens/s, Avg generation throughput: 334.8 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:28:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1015.5 tokens/s, Avg generation throughput: 484.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:28:23 [loggers.py:273] Engine 000: Avg prompt throughput: 155.5 tokens/s, Avg generation throughput: 528.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:28:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 502.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:28:43 [loggers.py:273] Engine 000: Avg prompt throughput: 127.1 tokens/s, Avg generation throughput: 489.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:28:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 496.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:29:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 492.1 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:29:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 473.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:29:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 473.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:29:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 441.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:29:43 [loggers.py:273] Engine 000: Avg prompt throughput: 743.6 tokens/s, Avg generation throughput: 407.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:29:53 [loggers.py:273] Engine 000: Avg prompt throughput: 795.4 tokens/s, Avg generation throughput: 413.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:30:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2430.3 tokens/s, Avg generation throughput: 334.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:30:13 [loggers.py:273] Engine 000: Avg prompt throughput: 779.6 tokens/s, Avg generation throughput: 428.4 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:30:23 [loggers.py:273] Engine 000: Avg prompt throughput: 3082.5 tokens/s, Avg generation throughput: 336.3 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:30:33 [loggers.py:273] Engine 000: Avg prompt throughput: 584.5 tokens/s, Avg generation throughput: 528.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:30:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 533.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:30:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 520.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:31:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 495.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:31:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 469.5 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:31:23 [loggers.py:273] Engine 000: Avg prompt throughput: 163.8 tokens/s, Avg generation throughput: 490.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:31:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 476.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:31:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 467.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:31:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 463.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:32:03 [loggers.py:273] Engine 000: Avg prompt throughput: 182.1 tokens/s, Avg generation throughput: 415.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:32:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1232.2 tokens/s, Avg generation throughput: 372.0 tokens/s, Running: 46 reqs, Waiting: 16 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:32:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2443.2 tokens/s, Avg generation throughput: 352.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:32:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1350.2 tokens/s, Avg generation throughput: 419.1 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 94.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:32:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2551.7 tokens/s, Avg generation throughput: 349.1 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:32:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 530.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:33:03 [loggers.py:273] Engine 000: Avg prompt throughput: 258.8 tokens/s, Avg generation throughput: 489.9 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:33:13 [loggers.py:273] Engine 000: Avg prompt throughput: 122.9 tokens/s, Avg generation throughput: 470.5 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:33:23 [loggers.py:273] Engine 000: Avg prompt throughput: 446.6 tokens/s, Avg generation throughput: 485.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:33:33 [loggers.py:273] Engine 000: Avg prompt throughput: 141.4 tokens/s, Avg generation throughput: 489.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:33:43 [loggers.py:273] Engine 000: Avg prompt throughput: 682.3 tokens/s, Avg generation throughput: 444.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:33:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 499.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:34:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 485.8 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:34:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 445.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:34:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 467.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:34:33 [loggers.py:273] Engine 000: Avg prompt throughput: 778.0 tokens/s, Avg generation throughput: 346.1 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:34:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2074.8 tokens/s, Avg generation throughput: 373.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:34:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1707.0 tokens/s, Avg generation throughput: 383.4 tokens/s, Running: 49 reqs, Waiting: 12 reqs, GPU KV cache usage: 91.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:35:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2684.7 tokens/s, Avg generation throughput: 367.3 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:35:13 [loggers.py:273] Engine 000: Avg prompt throughput: 157.0 tokens/s, Avg generation throughput: 529.7 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:35:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 533.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:35:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 489.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:35:43 [loggers.py:273] Engine 000: Avg prompt throughput: 141.3 tokens/s, Avg generation throughput: 491.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:35:53 [loggers.py:273] Engine 000: Avg prompt throughput: 150.3 tokens/s, Avg generation throughput: 503.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:36:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 475.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:36:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:36:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 476.3 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:36:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 449.6 tokens/s, Running: 43 reqs, Waiting: 20 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:36:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 439.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:36:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2046.3 tokens/s, Avg generation throughput: 322.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:37:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1301.2 tokens/s, Avg generation throughput: 403.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:37:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2885.7 tokens/s, Avg generation throughput: 319.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:37:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1734.5 tokens/s, Avg generation throughput: 416.9 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:37:33 [loggers.py:273] Engine 000: Avg prompt throughput: 142.4 tokens/s, Avg generation throughput: 520.5 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:37:43 [loggers.py:273] Engine 000: Avg prompt throughput: 154.1 tokens/s, Avg generation throughput: 518.6 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:37:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 519.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:38:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 508.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:38:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 462.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:38:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 479.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:38:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 475.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:38:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 473.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:38:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 463.0 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:39:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 435.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:39:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1897.9 tokens/s, Avg generation throughput: 325.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:39:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1024.8 tokens/s, Avg generation throughput: 423.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:39:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2943.9 tokens/s, Avg generation throughput: 322.3 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:39:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2025.6 tokens/s, Avg generation throughput: 415.2 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:39:53 [loggers.py:273] Engine 000: Avg prompt throughput: 624.3 tokens/s, Avg generation throughput: 484.9 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:40:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 541.9 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:40:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 502.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:40:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 499.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:40:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 501.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:40:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 487.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:40:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 472.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:41:03 [loggers.py:273] Engine 000: Avg prompt throughput: 148.4 tokens/s, Avg generation throughput: 461.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:41:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 469.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:41:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 449.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:41:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1809.1 tokens/s, Avg generation throughput: 361.6 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:41:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1316.2 tokens/s, Avg generation throughput: 414.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:41:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2791.9 tokens/s, Avg generation throughput: 334.0 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:42:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1646.9 tokens/s, Avg generation throughput: 445.3 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:42:13 [loggers.py:273] Engine 000: Avg prompt throughput: 424.7 tokens/s, Avg generation throughput: 512.3 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:42:23 [loggers.py:273] Engine 000: Avg prompt throughput: 140.4 tokens/s, Avg generation throughput: 508.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:42:33 [loggers.py:273] Engine 000: Avg prompt throughput: 311.2 tokens/s, Avg generation throughput: 482.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:42:43 [loggers.py:273] Engine 000: Avg prompt throughput: 184.5 tokens/s, Avg generation throughput: 503.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:42:53 [loggers.py:273] Engine 000: Avg prompt throughput: 147.3 tokens/s, Avg generation throughput: 484.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:43:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 506.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:43:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 475.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:43:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 476.3 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:43:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 455.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:43:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 437.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:43:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1631.4 tokens/s, Avg generation throughput: 329.5 tokens/s, Running: 46 reqs, Waiting: 16 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:44:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1268.3 tokens/s, Avg generation throughput: 421.2 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:44:13 [loggers.py:273] Engine 000: Avg prompt throughput: 3722.3 tokens/s, Avg generation throughput: 285.3 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:44:23 [loggers.py:273] Engine 000: Avg prompt throughput: 974.5 tokens/s, Avg generation throughput: 476.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:44:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 530.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:44:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 494.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:44:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 483.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:45:03 [loggers.py:273] Engine 000: Avg prompt throughput: 323.0 tokens/s, Avg generation throughput: 475.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:45:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 502.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:45:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:45:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:45:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:45:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 444.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:46:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 436.8 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:46:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2157.3 tokens/s, Avg generation throughput: 280.5 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:46:23 [loggers.py:273] Engine 000: Avg prompt throughput: 660.5 tokens/s, Avg generation throughput: 451.3 tokens/s, Running: 47 reqs, Waiting: 14 reqs, GPU KV cache usage: 95.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:46:33 [loggers.py:273] Engine 000: Avg prompt throughput: 4412.4 tokens/s, Avg generation throughput: 239.5 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:46:43 [loggers.py:273] Engine 000: Avg prompt throughput: 833.5 tokens/s, Avg generation throughput: 531.9 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:46:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 535.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:47:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 521.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:47:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 499.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:47:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 503.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:47:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:47:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 484.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:47:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 482.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:48:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 472.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:48:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 453.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:48:23 [loggers.py:273] Engine 000: Avg prompt throughput: 270.9 tokens/s, Avg generation throughput: 395.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:48:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2265.8 tokens/s, Avg generation throughput: 335.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:48:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1434.3 tokens/s, Avg generation throughput: 386.7 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 93.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:48:53 [loggers.py:273] Engine 000: Avg prompt throughput: 3966.6 tokens/s, Avg generation throughput: 293.3 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:49:03 [loggers.py:273] Engine 000: Avg prompt throughput: 411.5 tokens/s, Avg generation throughput: 508.3 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:49:13 [loggers.py:273] Engine 000: Avg prompt throughput: 141.0 tokens/s, Avg generation throughput: 520.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:49:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 509.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:49:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:49:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 510.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:49:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 474.3 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:50:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 487.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:50:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 438.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:50:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 459.7 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:50:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 457.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:50:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 410.2 tokens/s, Running: 43 reqs, Waiting: 19 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:50:53 [loggers.py:273] Engine 000: Avg prompt throughput: 3197.3 tokens/s, Avg generation throughput: 283.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:51:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.9 tokens/s, Avg generation throughput: 398.3 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 94.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:51:13 [loggers.py:273] Engine 000: Avg prompt throughput: 3724.6 tokens/s, Avg generation throughput: 310.7 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:51:23 [loggers.py:273] Engine 000: Avg prompt throughput: 413.7 tokens/s, Avg generation throughput: 528.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:51:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 522.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:51:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 524.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:51:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 484.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:52:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 512.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:52:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 486.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:52:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 477.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:52:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 477.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:52:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 466.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:52:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:53:03 [loggers.py:273] Engine 000: Avg prompt throughput: 267.5 tokens/s, Avg generation throughput: 385.9 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:53:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2726.2 tokens/s, Avg generation throughput: 317.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:53:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2404.3 tokens/s, Avg generation throughput: 357.6 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 92.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:53:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2491.9 tokens/s, Avg generation throughput: 397.5 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:53:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 550.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:53:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 522.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:54:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 491.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:54:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 510.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:54:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 509.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:54:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 495.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:54:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 447.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:54:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 474.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 08:55:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 461.9 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:55:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.5 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:55:23 [loggers.py:273] Engine 000: Avg prompt throughput: 298.2 tokens/s, Avg generation throughput: 364.9 tokens/s, Running: 44 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:55:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2737.6 tokens/s, Avg generation throughput: 321.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:55:43 [loggers.py:273] Engine 000: Avg prompt throughput: 3521.1 tokens/s, Avg generation throughput: 280.6 tokens/s, Running: 55 reqs, Waiting: 6 reqs, GPU KV cache usage: 95.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:55:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1691.6 tokens/s, Avg generation throughput: 468.3 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:56:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1704.2 tokens/s, Avg generation throughput: 445.1 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:56:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2151.1 tokens/s, Avg generation throughput: 427.3 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:56:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2528.3 tokens/s, Avg generation throughput: 391.9 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:56:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1201.4 tokens/s, Avg generation throughput: 465.0 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:56:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1558.2 tokens/s, Avg generation throughput: 429.4 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:56:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1093.3 tokens/s, Avg generation throughput: 467.4 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:57:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2254.3 tokens/s, Avg generation throughput: 387.8 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:57:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2033.0 tokens/s, Avg generation throughput: 390.6 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:57:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1119.9 tokens/s, Avg generation throughput: 463.0 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:57:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1331.2 tokens/s, Avg generation throughput: 435.4 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:57:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1909.3 tokens/s, Avg generation throughput: 420.5 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:57:53 [loggers.py:273] Engine 000: Avg prompt throughput: 971.8 tokens/s, Avg generation throughput: 438.9 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:58:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1285.0 tokens/s, Avg generation throughput: 452.0 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:58:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1861.7 tokens/s, Avg generation throughput: 393.8 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:58:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1542.3 tokens/s, Avg generation throughput: 457.9 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:58:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1799.3 tokens/s, Avg generation throughput: 424.7 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:58:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2499.2 tokens/s, Avg generation throughput: 389.8 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:58:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1681.2 tokens/s, Avg generation throughput: 426.6 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:59:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1933.8 tokens/s, Avg generation throughput: 430.0 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:59:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1416.5 tokens/s, Avg generation throughput: 446.0 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:59:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2167.7 tokens/s, Avg generation throughput: 412.4 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:59:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2246.1 tokens/s, Avg generation throughput: 381.3 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:59:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1460.9 tokens/s, Avg generation throughput: 458.1 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 08:59:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1905.5 tokens/s, Avg generation throughput: 432.0 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:00:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2281.6 tokens/s, Avg generation throughput: 390.3 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:00:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1790.4 tokens/s, Avg generation throughput: 431.4 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:00:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1908.6 tokens/s, Avg generation throughput: 420.6 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:00:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1229.0 tokens/s, Avg generation throughput: 469.5 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:00:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1305.4 tokens/s, Avg generation throughput: 449.2 tokens/s, Running: 54 reqs, Waiting: 7 reqs, GPU KV cache usage: 95.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:00:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1464.3 tokens/s, Avg generation throughput: 449.1 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:01:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2338.7 tokens/s, Avg generation throughput: 377.4 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:01:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1364.6 tokens/s, Avg generation throughput: 464.1 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:01:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1228.0 tokens/s, Avg generation throughput: 455.6 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:01:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2101.5 tokens/s, Avg generation throughput: 403.1 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:01:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1902.6 tokens/s, Avg generation throughput: 424.5 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:01:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1408.3 tokens/s, Avg generation throughput: 451.5 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:02:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1230.0 tokens/s, Avg generation throughput: 464.4 tokens/s, Running: 53 reqs, Waiting: 7 reqs, GPU KV cache usage: 94.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:02:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1213.0 tokens/s, Avg generation throughput: 456.8 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:02:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1689.0 tokens/s, Avg generation throughput: 421.0 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:02:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1774.3 tokens/s, Avg generation throughput: 430.6 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:02:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1857.0 tokens/s, Avg generation throughput: 413.0 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:02:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1529.4 tokens/s, Avg generation throughput: 457.9 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:03:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1809.5 tokens/s, Avg generation throughput: 446.6 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:03:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2051.2 tokens/s, Avg generation throughput: 413.9 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:03:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1153.1 tokens/s, Avg generation throughput: 480.9 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:03:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2037.4 tokens/s, Avg generation throughput: 420.8 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:03:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1782.7 tokens/s, Avg generation throughput: 441.1 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:03:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2130.3 tokens/s, Avg generation throughput: 412.6 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:04:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1672.7 tokens/s, Avg generation throughput: 435.3 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:04:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1724.2 tokens/s, Avg generation throughput: 430.4 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:04:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1093.5 tokens/s, Avg generation throughput: 482.5 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:04:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1167.7 tokens/s, Avg generation throughput: 476.8 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:04:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2132.4 tokens/s, Avg generation throughput: 396.3 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:04:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1693.7 tokens/s, Avg generation throughput: 428.0 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:05:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1572.0 tokens/s, Avg generation throughput: 435.8 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:05:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2790.6 tokens/s, Avg generation throughput: 342.0 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:05:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2918.1 tokens/s, Avg generation throughput: 306.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:05:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2997.9 tokens/s, Avg generation throughput: 330.2 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:05:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1968.0 tokens/s, Avg generation throughput: 350.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:05:53 [loggers.py:273] Engine 000: Avg prompt throughput: 4039.3 tokens/s, Avg generation throughput: 245.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:06:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1770.2 tokens/s, Avg generation throughput: 340.6 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:06:13 [loggers.py:273] Engine 000: Avg prompt throughput: 3577.5 tokens/s, Avg generation throughput: 245.3 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:06:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2363.1 tokens/s, Avg generation throughput: 319.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:06:33 [loggers.py:273] Engine 000: Avg prompt throughput: 3436.3 tokens/s, Avg generation throughput: 253.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:06:43 [loggers.py:273] Engine 000: Avg prompt throughput: 3076.7 tokens/s, Avg generation throughput: 281.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:06:53 [loggers.py:273] Engine 000: Avg prompt throughput: 3330.6 tokens/s, Avg generation throughput: 262.2 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:07:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2625.2 tokens/s, Avg generation throughput: 292.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:07:13 [loggers.py:273] Engine 000: Avg prompt throughput: 3086.1 tokens/s, Avg generation throughput: 270.1 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:07:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2857.6 tokens/s, Avg generation throughput: 270.8 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:07:33 [loggers.py:273] Engine 000: Avg prompt throughput: 3081.1 tokens/s, Avg generation throughput: 267.1 tokens/s, Running: 42 reqs, Waiting: 21 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:07:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2594.4 tokens/s, Avg generation throughput: 270.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:07:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2880.0 tokens/s, Avg generation throughput: 301.7 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:08:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2981.1 tokens/s, Avg generation throughput: 272.3 tokens/s, Running: 43 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:08:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2362.5 tokens/s, Avg generation throughput: 309.6 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:08:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2188.6 tokens/s, Avg generation throughput: 300.6 tokens/s, Running: 41 reqs, Waiting: 22 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:08:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2903.8 tokens/s, Avg generation throughput: 265.1 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:08:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2805.0 tokens/s, Avg generation throughput: 261.9 tokens/s, Running: 42 reqs, Waiting: 21 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:08:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2400.4 tokens/s, Avg generation throughput: 304.8 tokens/s, Running: 42 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:09:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2681.6 tokens/s, Avg generation throughput: 291.2 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:09:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2266.6 tokens/s, Avg generation throughput: 301.6 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:09:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1678.6 tokens/s, Avg generation throughput: 331.0 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:09:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2009.8 tokens/s, Avg generation throughput: 313.2 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:09:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2206.0 tokens/s, Avg generation throughput: 321.5 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:09:53 [loggers.py:273] Engine 000: Avg prompt throughput: 3093.5 tokens/s, Avg generation throughput: 257.6 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:10:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2494.9 tokens/s, Avg generation throughput: 276.4 tokens/s, Running: 41 reqs, Waiting: 22 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:10:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2933.4 tokens/s, Avg generation throughput: 255.3 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:10:23 [loggers.py:273] Engine 000: Avg prompt throughput: 3336.7 tokens/s, Avg generation throughput: 254.8 tokens/s, Running: 42 reqs, Waiting: 21 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:10:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2648.4 tokens/s, Avg generation throughput: 291.5 tokens/s, Running: 44 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:10:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2843.7 tokens/s, Avg generation throughput: 315.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:10:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2324.0 tokens/s, Avg generation throughput: 354.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:11:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2085.4 tokens/s, Avg generation throughput: 352.9 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:11:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1596.4 tokens/s, Avg generation throughput: 391.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:11:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2109.8 tokens/s, Avg generation throughput: 364.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:11:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2446.7 tokens/s, Avg generation throughput: 349.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:11:43 [loggers.py:273] Engine 000: Avg prompt throughput: 2156.9 tokens/s, Avg generation throughput: 372.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:11:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1737.3 tokens/s, Avg generation throughput: 396.0 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:12:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1973.7 tokens/s, Avg generation throughput: 384.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:12:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1639.0 tokens/s, Avg generation throughput: 389.8 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:12:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1952.3 tokens/s, Avg generation throughput: 402.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:12:33 [loggers.py:273] Engine 000: Avg prompt throughput: 951.5 tokens/s, Avg generation throughput: 437.5 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:12:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1874.2 tokens/s, Avg generation throughput: 400.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:12:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1213.5 tokens/s, Avg generation throughput: 425.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:13:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1284.4 tokens/s, Avg generation throughput: 418.8 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:13:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2249.7 tokens/s, Avg generation throughput: 365.6 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:13:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1577.7 tokens/s, Avg generation throughput: 432.3 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:13:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1116.4 tokens/s, Avg generation throughput: 439.1 tokens/s, Running: 48 reqs, Waiting: 14 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:13:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1761.8 tokens/s, Avg generation throughput: 398.8 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:13:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1317.0 tokens/s, Avg generation throughput: 431.4 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:14:03 [loggers.py:273] Engine 000: Avg prompt throughput: 939.9 tokens/s, Avg generation throughput: 435.7 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:14:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1667.2 tokens/s, Avg generation throughput: 395.0 tokens/s, Running: 49 reqs, Waiting: 13 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:14:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1597.3 tokens/s, Avg generation throughput: 424.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:14:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2351.8 tokens/s, Avg generation throughput: 369.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:14:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1947.3 tokens/s, Avg generation throughput: 363.2 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:14:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2066.1 tokens/s, Avg generation throughput: 377.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:15:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1269.5 tokens/s, Avg generation throughput: 412.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:15:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2034.6 tokens/s, Avg generation throughput: 352.3 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:15:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1723.7 tokens/s, Avg generation throughput: 401.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:15:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1627.6 tokens/s, Avg generation throughput: 412.6 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:15:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1835.5 tokens/s, Avg generation throughput: 392.6 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:15:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2095.2 tokens/s, Avg generation throughput: 373.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:16:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1155.6 tokens/s, Avg generation throughput: 438.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:16:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1892.6 tokens/s, Avg generation throughput: 396.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:16:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2107.8 tokens/s, Avg generation throughput: 372.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:16:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1519.0 tokens/s, Avg generation throughput: 421.3 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:16:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1759.4 tokens/s, Avg generation throughput: 412.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:16:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2120.2 tokens/s, Avg generation throughput: 374.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:17:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1792.2 tokens/s, Avg generation throughput: 403.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:17:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1154.2 tokens/s, Avg generation throughput: 421.6 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:17:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2333.0 tokens/s, Avg generation throughput: 355.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:17:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1144.2 tokens/s, Avg generation throughput: 439.0 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:17:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1777.8 tokens/s, Avg generation throughput: 402.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:17:53 [loggers.py:273] Engine 000: Avg prompt throughput: 971.5 tokens/s, Avg generation throughput: 413.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:18:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1707.8 tokens/s, Avg generation throughput: 400.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:18:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1211.2 tokens/s, Avg generation throughput: 438.0 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:18:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1450.3 tokens/s, Avg generation throughput: 394.4 tokens/s, Running: 48 reqs, Waiting: 14 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:18:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1433.9 tokens/s, Avg generation throughput: 425.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:18:43 [loggers.py:273] Engine 000: Avg prompt throughput: 964.6 tokens/s, Avg generation throughput: 427.8 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:18:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2719.1 tokens/s, Avg generation throughput: 325.5 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:19:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2155.0 tokens/s, Avg generation throughput: 339.7 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:19:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2779.9 tokens/s, Avg generation throughput: 344.6 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:19:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1334.3 tokens/s, Avg generation throughput: 428.3 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:19:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2241.3 tokens/s, Avg generation throughput: 367.9 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:19:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1822.5 tokens/s, Avg generation throughput: 402.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:19:53 [loggers.py:273] Engine 000: Avg prompt throughput: 3033.0 tokens/s, Avg generation throughput: 346.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:20:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1913.7 tokens/s, Avg generation throughput: 412.0 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:20:13 [loggers.py:273] Engine 000: Avg prompt throughput: 983.7 tokens/s, Avg generation throughput: 430.2 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:20:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1068.3 tokens/s, Avg generation throughput: 456.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:20:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1125.3 tokens/s, Avg generation throughput: 432.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:20:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1787.1 tokens/s, Avg generation throughput: 396.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:20:53 [loggers.py:273] Engine 000: Avg prompt throughput: 826.9 tokens/s, Avg generation throughput: 449.7 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:21:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2112.5 tokens/s, Avg generation throughput: 382.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:21:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1477.6 tokens/s, Avg generation throughput: 400.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:21:23 [loggers.py:273] Engine 000: Avg prompt throughput: 868.1 tokens/s, Avg generation throughput: 429.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:21:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1736.5 tokens/s, Avg generation throughput: 375.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:21:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1940.8 tokens/s, Avg generation throughput: 367.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:21:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2393.8 tokens/s, Avg generation throughput: 339.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:22:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1513.9 tokens/s, Avg generation throughput: 421.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:22:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1768.4 tokens/s, Avg generation throughput: 412.5 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:22:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1585.7 tokens/s, Avg generation throughput: 394.0 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:22:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2301.2 tokens/s, Avg generation throughput: 383.5 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:22:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1992.0 tokens/s, Avg generation throughput: 402.1 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:22:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2230.5 tokens/s, Avg generation throughput: 370.2 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:23:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1753.1 tokens/s, Avg generation throughput: 392.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:23:13 [loggers.py:273] Engine 000: Avg prompt throughput: 2258.3 tokens/s, Avg generation throughput: 386.1 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:23:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1340.2 tokens/s, Avg generation throughput: 422.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:23:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1806.6 tokens/s, Avg generation throughput: 395.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:23:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1478.7 tokens/s, Avg generation throughput: 437.9 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:23:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2071.0 tokens/s, Avg generation throughput: 395.0 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:24:03 [loggers.py:273] Engine 000: Avg prompt throughput: 930.1 tokens/s, Avg generation throughput: 431.8 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:24:13 [loggers.py:273] Engine 000: Avg prompt throughput: 798.5 tokens/s, Avg generation throughput: 455.5 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:24:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1262.8 tokens/s, Avg generation throughput: 430.5 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:24:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1153.9 tokens/s, Avg generation throughput: 408.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:24:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1403.3 tokens/s, Avg generation throughput: 371.6 tokens/s, Running: 46 reqs, Waiting: 16 reqs, GPU KV cache usage: 94.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:24:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1422.7 tokens/s, Avg generation throughput: 405.3 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:25:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2396.9 tokens/s, Avg generation throughput: 350.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:25:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1418.7 tokens/s, Avg generation throughput: 413.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:25:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1676.6 tokens/s, Avg generation throughput: 396.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:25:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2309.6 tokens/s, Avg generation throughput: 361.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:25:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1632.5 tokens/s, Avg generation throughput: 419.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:25:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1793.2 tokens/s, Avg generation throughput: 384.4 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:26:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1619.4 tokens/s, Avg generation throughput: 410.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:26:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1466.7 tokens/s, Avg generation throughput: 410.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:26:23 [loggers.py:273] Engine 000: Avg prompt throughput: 931.1 tokens/s, Avg generation throughput: 432.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:26:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1629.2 tokens/s, Avg generation throughput: 413.9 tokens/s, Running: 48 reqs, Waiting: 14 reqs, GPU KV cache usage: 95.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:26:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1218.8 tokens/s, Avg generation throughput: 419.8 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:26:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1904.0 tokens/s, Avg generation throughput: 372.7 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:27:03 [loggers.py:273] Engine 000: Avg prompt throughput: 901.2 tokens/s, Avg generation throughput: 441.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:27:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1215.8 tokens/s, Avg generation throughput: 410.6 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:27:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1794.3 tokens/s, Avg generation throughput: 368.3 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:27:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2124.6 tokens/s, Avg generation throughput: 376.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:27:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1124.8 tokens/s, Avg generation throughput: 419.7 tokens/s, Running: 46 reqs, Waiting: 16 reqs, GPU KV cache usage: 95.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:27:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1168.8 tokens/s, Avg generation throughput: 402.1 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:28:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2498.2 tokens/s, Avg generation throughput: 359.5 tokens/s, Running: 49 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:28:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1119.2 tokens/s, Avg generation throughput: 425.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:28:23 [loggers.py:273] Engine 000: Avg prompt throughput: 376.2 tokens/s, Avg generation throughput: 408.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:28:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2085.0 tokens/s, Avg generation throughput: 341.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:28:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1901.9 tokens/s, Avg generation throughput: 337.3 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:28:53 [loggers.py:273] Engine 000: Avg prompt throughput: 2085.8 tokens/s, Avg generation throughput: 366.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:29:03 [loggers.py:273] Engine 000: Avg prompt throughput: 748.2 tokens/s, Avg generation throughput: 417.3 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:29:13 [loggers.py:273] Engine 000: Avg prompt throughput: 935.5 tokens/s, Avg generation throughput: 433.3 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:29:23 [loggers.py:273] Engine 000: Avg prompt throughput: 935.0 tokens/s, Avg generation throughput: 401.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:29:33 [loggers.py:273] Engine 000: Avg prompt throughput: 189.7 tokens/s, Avg generation throughput: 431.1 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:29:43 [loggers.py:273] Engine 000: Avg prompt throughput: 572.5 tokens/s, Avg generation throughput: 411.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:29:53 [loggers.py:273] Engine 000: Avg prompt throughput: 587.6 tokens/s, Avg generation throughput: 404.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:30:03 [loggers.py:273] Engine 000: Avg prompt throughput: 778.8 tokens/s, Avg generation throughput: 401.3 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:30:13 [loggers.py:273] Engine 000: Avg prompt throughput: 385.2 tokens/s, Avg generation throughput: 431.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:30:23 [loggers.py:273] Engine 000: Avg prompt throughput: 969.8 tokens/s, Avg generation throughput: 376.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:30:33 [loggers.py:273] Engine 000: Avg prompt throughput: 562.4 tokens/s, Avg generation throughput: 421.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:30:43 [loggers.py:273] Engine 000: Avg prompt throughput: 185.1 tokens/s, Avg generation throughput: 415.6 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:30:53 [loggers.py:273] Engine 000: Avg prompt throughput: 377.4 tokens/s, Avg generation throughput: 403.9 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:31:03 [loggers.py:273] Engine 000: Avg prompt throughput: 569.5 tokens/s, Avg generation throughput: 387.8 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:31:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1956.9 tokens/s, Avg generation throughput: 356.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:31:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1131.8 tokens/s, Avg generation throughput: 393.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:31:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1304.9 tokens/s, Avg generation throughput: 365.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:31:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1692.8 tokens/s, Avg generation throughput: 351.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:31:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1509.0 tokens/s, Avg generation throughput: 376.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:32:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1310.8 tokens/s, Avg generation throughput: 390.1 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 09:32:13 [loggers.py:273] Engine 000: Avg prompt throughput: 187.2 tokens/s, Avg generation throughput: 449.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:32:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1694.2 tokens/s, Avg generation throughput: 355.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:32:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1131.2 tokens/s, Avg generation throughput: 404.5 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:32:43 [loggers.py:273] Engine 000: Avg prompt throughput: 936.2 tokens/s, Avg generation throughput: 381.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:32:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1315.2 tokens/s, Avg generation throughput: 393.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:33:03 [loggers.py:273] Engine 000: Avg prompt throughput: 190.5 tokens/s, Avg generation throughput: 442.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:33:13 [loggers.py:273] Engine 000: Avg prompt throughput: 380.8 tokens/s, Avg generation throughput: 441.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:33:23 [loggers.py:273] Engine 000: Avg prompt throughput: 185.8 tokens/s, Avg generation throughput: 417.8 tokens/s, Running: 43 reqs, Waiting: 20 reqs, GPU KV cache usage: 95.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:33:33 [loggers.py:273] Engine 000: Avg prompt throughput: 386.3 tokens/s, Avg generation throughput: 395.5 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:33:43 [loggers.py:273] Engine 000: Avg prompt throughput: 798.2 tokens/s, Avg generation throughput: 399.7 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:33:53 [loggers.py:273] Engine 000: Avg prompt throughput: 193.4 tokens/s, Avg generation throughput: 435.5 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:34:03 [loggers.py:273] Engine 000: Avg prompt throughput: 188.5 tokens/s, Avg generation throughput: 410.3 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:34:13 [loggers.py:273] Engine 000: Avg prompt throughput: 786.0 tokens/s, Avg generation throughput: 376.6 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:34:23 [loggers.py:273] Engine 000: Avg prompt throughput: 978.3 tokens/s, Avg generation throughput: 389.6 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:34:33 [loggers.py:273] Engine 000: Avg prompt throughput: 748.5 tokens/s, Avg generation throughput: 400.0 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:34:43 [loggers.py:273] Engine 000: Avg prompt throughput: 956.9 tokens/s, Avg generation throughput: 369.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:34:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1004.7 tokens/s, Avg generation throughput: 390.5 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:35:03 [loggers.py:273] Engine 000: Avg prompt throughput: 791.5 tokens/s, Avg generation throughput: 407.9 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:35:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1902.2 tokens/s, Avg generation throughput: 338.0 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:35:23 [loggers.py:273] Engine 000: Avg prompt throughput: 941.3 tokens/s, Avg generation throughput: 408.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:35:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1128.0 tokens/s, Avg generation throughput: 375.9 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:35:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1703.7 tokens/s, Avg generation throughput: 380.1 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:35:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1525.7 tokens/s, Avg generation throughput: 367.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:36:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1312.5 tokens/s, Avg generation throughput: 397.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:36:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1876.0 tokens/s, Avg generation throughput: 354.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:36:23 [loggers.py:273] Engine 000: Avg prompt throughput: 379.5 tokens/s, Avg generation throughput: 451.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:36:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 453.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:36:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1128.7 tokens/s, Avg generation throughput: 370.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:36:53 [loggers.py:273] Engine 000: Avg prompt throughput: 973.7 tokens/s, Avg generation throughput: 392.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:37:03 [loggers.py:273] Engine 000: Avg prompt throughput: 965.0 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:37:13 [loggers.py:273] Engine 000: Avg prompt throughput: 390.7 tokens/s, Avg generation throughput: 432.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:37:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 434.0 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:37:33 [loggers.py:273] Engine 000: Avg prompt throughput: 593.0 tokens/s, Avg generation throughput: 408.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:37:43 [loggers.py:273] Engine 000: Avg prompt throughput: 195.3 tokens/s, Avg generation throughput: 437.5 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:37:53 [loggers.py:273] Engine 000: Avg prompt throughput: 197.1 tokens/s, Avg generation throughput: 416.8 tokens/s, Running: 43 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:38:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1174.6 tokens/s, Avg generation throughput: 373.5 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:38:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1151.2 tokens/s, Avg generation throughput: 371.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:38:23 [loggers.py:273] Engine 000: Avg prompt throughput: 563.1 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:38:33 [loggers.py:273] Engine 000: Avg prompt throughput: 2127.8 tokens/s, Avg generation throughput: 332.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 09:38:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 455.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:38:53 [loggers.py:273] Engine 000: Avg prompt throughput: 378.0 tokens/s, Avg generation throughput: 397.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:39:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1118.0 tokens/s, Avg generation throughput: 378.3 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:39:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1131.4 tokens/s, Avg generation throughput: 385.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:39:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1692.3 tokens/s, Avg generation throughput: 356.8 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:39:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1303.9 tokens/s, Avg generation throughput: 390.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:39:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1124.4 tokens/s, Avg generation throughput: 399.5 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:39:53 [loggers.py:273] Engine 000: Avg prompt throughput: 947.3 tokens/s, Avg generation throughput: 404.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:40:03 [loggers.py:273] Engine 000: Avg prompt throughput: 927.8 tokens/s, Avg generation throughput: 408.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:40:13 [loggers.py:273] Engine 000: Avg prompt throughput: 754.1 tokens/s, Avg generation throughput: 410.3 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:40:23 [loggers.py:273] Engine 000: Avg prompt throughput: 956.3 tokens/s, Avg generation throughput: 376.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:40:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1302.4 tokens/s, Avg generation throughput: 392.7 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:40:43 [loggers.py:273] Engine 000: Avg prompt throughput: 565.8 tokens/s, Avg generation throughput: 428.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:40:53 [loggers.py:273] Engine 000: Avg prompt throughput: 574.5 tokens/s, Avg generation throughput: 422.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:41:03 [loggers.py:273] Engine 000: Avg prompt throughput: 743.4 tokens/s, Avg generation throughput: 423.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:41:13 [loggers.py:273] Engine 000: Avg prompt throughput: 375.9 tokens/s, Avg generation throughput: 397.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 09:41:23 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 451.7 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:41:33 [loggers.py:273] Engine 000: Avg prompt throughput: 184.5 tokens/s, Avg generation throughput: 422.1 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:41:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 421.7 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:41:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1933.7 tokens/s, Avg generation throughput: 327.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:42:03 [loggers.py:273] Engine 000: Avg prompt throughput: 384.1 tokens/s, Avg generation throughput: 407.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:42:13 [loggers.py:273] Engine 000: Avg prompt throughput: 973.5 tokens/s, Avg generation throughput: 377.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:42:23 [loggers.py:273] Engine 000: Avg prompt throughput: 965.9 tokens/s, Avg generation throughput: 374.0 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:42:33 [loggers.py:273] Engine 000: Avg prompt throughput: 780.5 tokens/s, Avg generation throughput: 409.9 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:42:43 [loggers.py:273] Engine 000: Avg prompt throughput: 996.6 tokens/s, Avg generation throughput: 377.6 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:42:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1567.7 tokens/s, Avg generation throughput: 358.8 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:43:03 [loggers.py:273] Engine 000: Avg prompt throughput: 949.4 tokens/s, Avg generation throughput: 404.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:43:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1112.7 tokens/s, Avg generation throughput: 370.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:43:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1889.6 tokens/s, Avg generation throughput: 358.3 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:43:33 [loggers.py:273] Engine 000: Avg prompt throughput: 922.3 tokens/s, Avg generation throughput: 399.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:43:43 [loggers.py:273] Engine 000: Avg prompt throughput: 742.5 tokens/s, Avg generation throughput: 398.8 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:43:53 [loggers.py:273] Engine 000: Avg prompt throughput: 936.4 tokens/s, Avg generation throughput: 400.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:44:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1312.2 tokens/s, Avg generation throughput: 381.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:44:13 [loggers.py:273] Engine 000: Avg prompt throughput: 746.7 tokens/s, Avg generation throughput: 409.0 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:44:23 [loggers.py:273] Engine 000: Avg prompt throughput: 2659.1 tokens/s, Avg generation throughput: 304.7 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:44:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1154.9 tokens/s, Avg generation throughput: 397.3 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:44:43 [loggers.py:273] Engine 000: Avg prompt throughput: 764.5 tokens/s, Avg generation throughput: 420.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:44:53 [loggers.py:273] Engine 000: Avg prompt throughput: 562.4 tokens/s, Avg generation throughput: 430.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:45:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 440.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:45:13 [loggers.py:273] Engine 000: Avg prompt throughput: 187.0 tokens/s, Avg generation throughput: 412.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:45:23 [loggers.py:273] Engine 000: Avg prompt throughput: 564.1 tokens/s, Avg generation throughput: 436.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 09:45:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 450.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 09:45:43 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 445.2 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:45:53 [loggers.py:273] Engine 000: Avg prompt throughput: 373.1 tokens/s, Avg generation throughput: 382.6 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:46:03 [loggers.py:273] Engine 000: Avg prompt throughput: 746.3 tokens/s, Avg generation throughput: 396.3 tokens/s, Running: 42 reqs, Waiting: 22 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:46:13 [loggers.py:273] Engine 000: Avg prompt throughput: 386.4 tokens/s, Avg generation throughput: 388.8 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:46:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1150.6 tokens/s, Avg generation throughput: 388.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:46:33 [loggers.py:273] Engine 000: Avg prompt throughput: 194.0 tokens/s, Avg generation throughput: 414.4 tokens/s, Running: 43 reqs, Waiting: 20 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:46:43 [loggers.py:273] Engine 000: Avg prompt throughput: 3280.8 tokens/s, Avg generation throughput: 279.7 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:46:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1878.7 tokens/s, Avg generation throughput: 354.5 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:47:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1714.7 tokens/s, Avg generation throughput: 356.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:47:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1307.7 tokens/s, Avg generation throughput: 393.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:47:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1315.3 tokens/s, Avg generation throughput: 398.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:47:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1496.3 tokens/s, Avg generation throughput: 384.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:47:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1893.5 tokens/s, Avg generation throughput: 367.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:47:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1337.8 tokens/s, Avg generation throughput: 395.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:48:03 [loggers.py:273] Engine 000: Avg prompt throughput: 379.4 tokens/s, Avg generation throughput: 442.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:48:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 427.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:48:23 [loggers.py:273] Engine 000: Avg prompt throughput: 767.1 tokens/s, Avg generation throughput: 399.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:48:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1132.6 tokens/s, Avg generation throughput: 384.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:48:43 [loggers.py:273] Engine 000: Avg prompt throughput: 572.4 tokens/s, Avg generation throughput: 422.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:48:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1131.6 tokens/s, Avg generation throughput: 398.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:49:03 [loggers.py:273] Engine 000: Avg prompt throughput: 184.2 tokens/s, Avg generation throughput: 451.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:49:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1568.4 tokens/s, Avg generation throughput: 333.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:49:23 [loggers.py:273] Engine 000: Avg prompt throughput: 193.4 tokens/s, Avg generation throughput: 422.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:49:33 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 428.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:49:43 [loggers.py:273] Engine 000: Avg prompt throughput: 183.9 tokens/s, Avg generation throughput: 411.9 tokens/s, Running: 42 reqs, Waiting: 21 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:49:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1146.7 tokens/s, Avg generation throughput: 382.0 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:50:03 [loggers.py:273] Engine 000: Avg prompt throughput: 991.4 tokens/s, Avg generation throughput: 383.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:50:13 [loggers.py:273] Engine 000: Avg prompt throughput: 976.8 tokens/s, Avg generation throughput: 380.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:50:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1318.1 tokens/s, Avg generation throughput: 368.1 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:50:33 [loggers.py:273] Engine 000: Avg prompt throughput: 774.4 tokens/s, Avg generation throughput: 382.0 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:50:43 [loggers.py:273] Engine 000: Avg prompt throughput: 932.9 tokens/s, Avg generation throughput: 391.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:50:53 [loggers.py:273] Engine 000: Avg prompt throughput: 1703.0 tokens/s, Avg generation throughput: 354.5 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:51:03 [loggers.py:273] Engine 000: Avg prompt throughput: 2065.9 tokens/s, Avg generation throughput: 350.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:51:13 [loggers.py:273] Engine 000: Avg prompt throughput: 931.3 tokens/s, Avg generation throughput: 418.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:51:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1316.8 tokens/s, Avg generation throughput: 387.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:51:33 [loggers.py:273] Engine 000: Avg prompt throughput: 1126.8 tokens/s, Avg generation throughput: 408.3 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:51:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1529.8 tokens/s, Avg generation throughput: 379.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:51:53 [loggers.py:273] Engine 000: Avg prompt throughput: 743.6 tokens/s, Avg generation throughput: 404.2 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:52:03 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 446.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:52:13 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:52:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1140.6 tokens/s, Avg generation throughput: 392.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:52:33 [loggers.py:273] Engine 000: Avg prompt throughput: 379.8 tokens/s, Avg generation throughput: 411.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:52:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1153.4 tokens/s, Avg generation throughput: 377.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 09:52:53 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 442.3 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:53:03 [loggers.py:273] Engine 000: Avg prompt throughput: 381.9 tokens/s, Avg generation throughput: 399.7 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:53:13 [loggers.py:273] Engine 000: Avg prompt throughput: 398.6 tokens/s, Avg generation throughput: 383.8 tokens/s, Running: 43 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:53:23 [loggers.py:273] Engine 000: Avg prompt throughput: 1354.1 tokens/s, Avg generation throughput: 382.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:53:33 [loggers.py:273] Engine 000: Avg prompt throughput: 578.0 tokens/s, Avg generation throughput: 408.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:53:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1212.9 tokens/s, Avg generation throughput: 380.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:53:53 [loggers.py:273] Engine 000: Avg prompt throughput: 954.7 tokens/s, Avg generation throughput: 374.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:54:03 [loggers.py:273] Engine 000: Avg prompt throughput: 1697.1 tokens/s, Avg generation throughput: 344.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:54:13 [loggers.py:273] Engine 000: Avg prompt throughput: 1140.1 tokens/s, Avg generation throughput: 399.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:54:23 [loggers.py:273] Engine 000: Avg prompt throughput: 772.8 tokens/s, Avg generation throughput: 413.3 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:54:33 [loggers.py:273] Engine 000: Avg prompt throughput: 750.9 tokens/s, Avg generation throughput: 405.8 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:54:43 [loggers.py:273] Engine 000: Avg prompt throughput: 1308.9 tokens/s, Avg generation throughput: 379.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:54:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1114.0 tokens/s, Avg generation throughput: 382.2 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:55:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1115.7 tokens/s, Avg generation throughput: 373.6 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:55:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1311.3 tokens/s, Avg generation throughput: 397.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:55:24 [loggers.py:273] Engine 000: Avg prompt throughput: 379.7 tokens/s, Avg generation throughput: 427.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:55:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1125.5 tokens/s, Avg generation throughput: 397.3 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:55:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1125.1 tokens/s, Avg generation throughput: 377.9 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:55:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1722.5 tokens/s, Avg generation throughput: 362.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:56:04 [loggers.py:273] Engine 000: Avg prompt throughput: 189.6 tokens/s, Avg generation throughput: 460.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:56:14 [loggers.py:273] Engine 000: Avg prompt throughput: 774.9 tokens/s, Avg generation throughput: 409.6 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:56:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 420.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:56:34 [loggers.py:273] Engine 000: Avg prompt throughput: 186.5 tokens/s, Avg generation throughput: 440.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:56:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 428.6 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:56:54 [loggers.py:273] Engine 000: Avg prompt throughput: 583.9 tokens/s, Avg generation throughput: 395.1 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:57:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1143.0 tokens/s, Avg generation throughput: 373.3 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:57:14 [loggers.py:273] Engine 000: Avg prompt throughput: 967.6 tokens/s, Avg generation throughput: 409.0 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:57:24 [loggers.py:273] Engine 000: Avg prompt throughput: 606.9 tokens/s, Avg generation throughput: 400.1 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:57:34 [loggers.py:273] Engine 000: Avg prompt throughput: 982.2 tokens/s, Avg generation throughput: 386.5 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:57:44 [loggers.py:273] Engine 000: Avg prompt throughput: 952.5 tokens/s, Avg generation throughput: 395.4 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:57:54 [loggers.py:273] Engine 000: Avg prompt throughput: 931.2 tokens/s, Avg generation throughput: 391.1 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:58:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1509.6 tokens/s, Avg generation throughput: 358.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:58:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2464.2 tokens/s, Avg generation throughput: 326.9 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:58:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1496.3 tokens/s, Avg generation throughput: 365.1 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:58:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1126.0 tokens/s, Avg generation throughput: 410.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:58:44 [loggers.py:273] Engine 000: Avg prompt throughput: 564.6 tokens/s, Avg generation throughput: 441.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:58:54 [loggers.py:273] Engine 000: Avg prompt throughput: 184.3 tokens/s, Avg generation throughput: 430.1 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:59:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1334.7 tokens/s, Avg generation throughput: 381.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:59:14 [loggers.py:273] Engine 000: Avg prompt throughput: 395.2 tokens/s, Avg generation throughput: 420.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:59:24 [loggers.py:273] Engine 000: Avg prompt throughput: 944.0 tokens/s, Avg generation throughput: 388.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:59:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1900.4 tokens/s, Avg generation throughput: 355.6 tokens/s, Running: 46 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:59:44 [loggers.py:273] Engine 000: Avg prompt throughput: 373.7 tokens/s, Avg generation throughput: 446.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 09:59:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 434.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:00:04 [loggers.py:273] Engine 000: Avg prompt throughput: 377.4 tokens/s, Avg generation throughput: 412.1 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:00:14 [loggers.py:273] Engine 000: Avg prompt throughput: 788.9 tokens/s, Avg generation throughput: 403.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:00:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 448.5 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:00:34 [loggers.py:273] Engine 000: Avg prompt throughput: 597.5 tokens/s, Avg generation throughput: 355.7 tokens/s, Running: 42 reqs, Waiting: 20 reqs, GPU KV cache usage: 94.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:00:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1557.1 tokens/s, Avg generation throughput: 374.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:00:54 [loggers.py:273] Engine 000: Avg prompt throughput: 578.2 tokens/s, Avg generation throughput: 402.5 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:01:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 421.4 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:01:14 [loggers.py:273] Engine 000: Avg prompt throughput: 200.9 tokens/s, Avg generation throughput: 429.8 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:01:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1524.4 tokens/s, Avg generation throughput: 350.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:01:34 [loggers.py:273] Engine 000: Avg prompt throughput: 188.8 tokens/s, Avg generation throughput: 439.9 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:01:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1344.8 tokens/s, Avg generation throughput: 379.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:01:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1806.8 tokens/s, Avg generation throughput: 329.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:02:04 [loggers.py:273] Engine 000: Avg prompt throughput: 934.6 tokens/s, Avg generation throughput: 396.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:02:14 [loggers.py:273] Engine 000: Avg prompt throughput: 192.6 tokens/s, Avg generation throughput: 434.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:02:24 [loggers.py:273] Engine 000: Avg prompt throughput: 372.3 tokens/s, Avg generation throughput: 407.5 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:02:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1135.0 tokens/s, Avg generation throughput: 375.1 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:02:44 [loggers.py:273] Engine 000: Avg prompt throughput: 562.1 tokens/s, Avg generation throughput: 392.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:02:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1139.9 tokens/s, Avg generation throughput: 385.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:03:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2036.9 tokens/s, Avg generation throughput: 334.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:03:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2247.9 tokens/s, Avg generation throughput: 326.4 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:03:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1314.8 tokens/s, Avg generation throughput: 393.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:03:34 [loggers.py:273] Engine 000: Avg prompt throughput: 958.1 tokens/s, Avg generation throughput: 391.7 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:03:44 [loggers.py:273] Engine 000: Avg prompt throughput: 579.5 tokens/s, Avg generation throughput: 422.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:03:54 [loggers.py:273] Engine 000: Avg prompt throughput: 986.6 tokens/s, Avg generation throughput: 399.6 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:04:04 [loggers.py:273] Engine 000: Avg prompt throughput: 386.4 tokens/s, Avg generation throughput: 418.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:04:14 [loggers.py:273] Engine 000: Avg prompt throughput: 945.5 tokens/s, Avg generation throughput: 401.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:04:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1495.9 tokens/s, Avg generation throughput: 372.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:04:34 [loggers.py:273] Engine 000: Avg prompt throughput: 183.2 tokens/s, Avg generation throughput: 460.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:04:44 [loggers.py:273] Engine 000: Avg prompt throughput: 371.4 tokens/s, Avg generation throughput: 422.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:04:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 431.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:05:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 429.9 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:05:14 [loggers.py:273] Engine 000: Avg prompt throughput: 185.9 tokens/s, Avg generation throughput: 416.9 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:05:24 [loggers.py:273] Engine 000: Avg prompt throughput: 981.7 tokens/s, Avg generation throughput: 366.4 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:05:34 [loggers.py:273] Engine 000: Avg prompt throughput: 776.2 tokens/s, Avg generation throughput: 404.4 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:05:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2084.5 tokens/s, Avg generation throughput: 335.7 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:05:54 [loggers.py:273] Engine 000: Avg prompt throughput: 397.3 tokens/s, Avg generation throughput: 399.6 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:06:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1403.6 tokens/s, Avg generation throughput: 386.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:06:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1297.1 tokens/s, Avg generation throughput: 370.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:06:24 [loggers.py:273] Engine 000: Avg prompt throughput: 743.5 tokens/s, Avg generation throughput: 400.3 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:06:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1696.3 tokens/s, Avg generation throughput: 361.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:06:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1326.9 tokens/s, Avg generation throughput: 389.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:06:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1881.9 tokens/s, Avg generation throughput: 366.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:07:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1310.0 tokens/s, Avg generation throughput: 387.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:07:14 [loggers.py:273] Engine 000: Avg prompt throughput: 370.9 tokens/s, Avg generation throughput: 421.1 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:07:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 456.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:07:34 [loggers.py:273] Engine 000: Avg prompt throughput: 186.6 tokens/s, Avg generation throughput: 402.5 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:07:44 [loggers.py:273] Engine 000: Avg prompt throughput: 757.3 tokens/s, Avg generation throughput: 382.9 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:07:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1716.8 tokens/s, Avg generation throughput: 357.9 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:08:04 [loggers.py:273] Engine 000: Avg prompt throughput: 953.5 tokens/s, Avg generation throughput: 394.9 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:08:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1382.4 tokens/s, Avg generation throughput: 398.3 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:08:24 [loggers.py:273] Engine 000: Avg prompt throughput: 402.2 tokens/s, Avg generation throughput: 408.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:08:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 445.1 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:08:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 421.2 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:08:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 416.7 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:09:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1739.7 tokens/s, Avg generation throughput: 339.3 tokens/s, Running: 44 reqs, Waiting: 18 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:09:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1855.8 tokens/s, Avg generation throughput: 365.8 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:09:24 [loggers.py:273] Engine 000: Avg prompt throughput: 959.3 tokens/s, Avg generation throughput: 393.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:09:34 [loggers.py:273] Engine 000: Avg prompt throughput: 185.1 tokens/s, Avg generation throughput: 424.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:09:44 [loggers.py:273] Engine 000: Avg prompt throughput: 185.5 tokens/s, Avg generation throughput: 427.8 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:09:54 [loggers.py:273] Engine 000: Avg prompt throughput: 372.8 tokens/s, Avg generation throughput: 406.6 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:10:04 [loggers.py:273] Engine 000: Avg prompt throughput: 750.6 tokens/s, Avg generation throughput: 399.6 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:10:14 [loggers.py:273] Engine 000: Avg prompt throughput: 944.8 tokens/s, Avg generation throughput: 393.1 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:10:24 [loggers.py:273] Engine 000: Avg prompt throughput: 2857.1 tokens/s, Avg generation throughput: 298.2 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:10:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1324.3 tokens/s, Avg generation throughput: 383.2 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:10:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1712.5 tokens/s, Avg generation throughput: 331.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:10:54 [loggers.py:273] Engine 000: Avg prompt throughput: 754.3 tokens/s, Avg generation throughput: 418.5 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:11:04 [loggers.py:273] Engine 000: Avg prompt throughput: 556.6 tokens/s, Avg generation throughput: 419.9 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:11:14 [loggers.py:273] Engine 000: Avg prompt throughput: 569.6 tokens/s, Avg generation throughput: 414.5 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:11:24 [loggers.py:273] Engine 000: Avg prompt throughput: 592.6 tokens/s, Avg generation throughput: 390.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:11:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1691.3 tokens/s, Avg generation throughput: 351.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:11:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1148.5 tokens/s, Avg generation throughput: 406.0 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:11:54 [loggers.py:273] Engine 000: Avg prompt throughput: 966.6 tokens/s, Avg generation throughput: 389.2 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:12:04 [loggers.py:273] Engine 000: Avg prompt throughput: 575.8 tokens/s, Avg generation throughput: 449.4 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:12:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 441.0 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:12:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 417.0 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:12:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 425.6 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:12:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 429.9 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:12:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1141.9 tokens/s, Avg generation throughput: 365.8 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:13:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1360.1 tokens/s, Avg generation throughput: 382.0 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:13:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1341.9 tokens/s, Avg generation throughput: 355.4 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:13:24 [loggers.py:273] Engine 000: Avg prompt throughput: 555.0 tokens/s, Avg generation throughput: 396.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:13:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1117.2 tokens/s, Avg generation throughput: 382.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:13:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1129.9 tokens/s, Avg generation throughput: 382.2 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:13:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1879.4 tokens/s, Avg generation throughput: 348.3 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:14:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2071.9 tokens/s, Avg generation throughput: 336.6 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:14:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1530.9 tokens/s, Avg generation throughput: 382.7 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:14:24 [loggers.py:273] Engine 000: Avg prompt throughput: 763.2 tokens/s, Avg generation throughput: 409.2 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:14:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1544.7 tokens/s, Avg generation throughput: 382.5 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:14:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 463.2 tokens/s, Running: 45 reqs, Waiting: 19 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:14:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 408.0 tokens/s, Running: 45 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:15:04 [loggers.py:273] Engine 000: Avg prompt throughput: 106.8 tokens/s, Avg generation throughput: 439.5 tokens/s, Running: 44 reqs, Waiting: 20 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:15:14 [loggers.py:273] Engine 000: Avg prompt throughput: 742.0 tokens/s, Avg generation throughput: 394.0 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:15:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1223.9 tokens/s, Avg generation throughput: 375.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:15:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1244.4 tokens/s, Avg generation throughput: 392.2 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:15:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2037.0 tokens/s, Avg generation throughput: 351.1 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:15:54 [loggers.py:273] Engine 000: Avg prompt throughput: 860.2 tokens/s, Avg generation throughput: 402.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:16:04 [loggers.py:273] Engine 000: Avg prompt throughput: 974.5 tokens/s, Avg generation throughput: 400.4 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:16:14 [loggers.py:273] Engine 000: Avg prompt throughput: 814.6 tokens/s, Avg generation throughput: 413.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:16:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1283.4 tokens/s, Avg generation throughput: 397.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:16:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1307.7 tokens/s, Avg generation throughput: 387.2 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:16:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2453.1 tokens/s, Avg generation throughput: 385.3 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:16:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2530.5 tokens/s, Avg generation throughput: 365.2 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:17:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2244.2 tokens/s, Avg generation throughput: 396.2 tokens/s, Running: 60 reqs, Waiting: 2 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:17:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2437.6 tokens/s, Avg generation throughput: 398.7 tokens/s, Running: 61 reqs, Waiting: 2 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:17:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1898.5 tokens/s, Avg generation throughput: 457.0 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:17:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1155.5 tokens/s, Avg generation throughput: 484.0 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:17:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1923.2 tokens/s, Avg generation throughput: 439.7 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:17:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1458.8 tokens/s, Avg generation throughput: 445.3 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:18:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1943.4 tokens/s, Avg generation throughput: 414.7 tokens/s, Running: 59 reqs, Waiting: 3 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:18:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1902.7 tokens/s, Avg generation throughput: 424.0 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:18:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1677.1 tokens/s, Avg generation throughput: 439.5 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:18:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1960.1 tokens/s, Avg generation throughput: 428.5 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:18:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2365.5 tokens/s, Avg generation throughput: 412.1 tokens/s, Running: 59 reqs, Waiting: 2 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:18:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2507.3 tokens/s, Avg generation throughput: 389.3 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:19:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1892.0 tokens/s, Avg generation throughput: 439.0 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:19:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1980.1 tokens/s, Avg generation throughput: 425.5 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:19:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1807.0 tokens/s, Avg generation throughput: 428.8 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:19:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1890.4 tokens/s, Avg generation throughput: 441.2 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:19:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1649.3 tokens/s, Avg generation throughput: 448.1 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:19:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2026.3 tokens/s, Avg generation throughput: 413.6 tokens/s, Running: 61 reqs, Waiting: 2 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:20:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2346.9 tokens/s, Avg generation throughput: 401.5 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:20:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1074.9 tokens/s, Avg generation throughput: 486.6 tokens/s, Running: 59 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:20:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1915.9 tokens/s, Avg generation throughput: 432.6 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:20:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1137.8 tokens/s, Avg generation throughput: 461.4 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:20:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1617.2 tokens/s, Avg generation throughput: 449.2 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:20:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1914.3 tokens/s, Avg generation throughput: 431.8 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:21:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1345.9 tokens/s, Avg generation throughput: 454.6 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:21:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1663.8 tokens/s, Avg generation throughput: 437.8 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:21:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1432.1 tokens/s, Avg generation throughput: 436.6 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:21:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2275.2 tokens/s, Avg generation throughput: 396.2 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:21:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2214.9 tokens/s, Avg generation throughput: 413.6 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:21:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1833.6 tokens/s, Avg generation throughput: 426.3 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:22:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2147.8 tokens/s, Avg generation throughput: 387.7 tokens/s, Running: 60 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:22:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2113.0 tokens/s, Avg generation throughput: 409.9 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:22:24 [loggers.py:273] Engine 000: Avg prompt throughput: 2195.8 tokens/s, Avg generation throughput: 396.4 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:22:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1954.3 tokens/s, Avg generation throughput: 422.8 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:22:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2402.1 tokens/s, Avg generation throughput: 394.1 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:22:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1614.8 tokens/s, Avg generation throughput: 461.8 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:23:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1886.7 tokens/s, Avg generation throughput: 412.7 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:23:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2353.0 tokens/s, Avg generation throughput: 398.5 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:23:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1656.4 tokens/s, Avg generation throughput: 446.6 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:23:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1336.9 tokens/s, Avg generation throughput: 462.8 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:23:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1097.5 tokens/s, Avg generation throughput: 481.9 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:23:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1639.2 tokens/s, Avg generation throughput: 448.0 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:24:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1977.5 tokens/s, Avg generation throughput: 421.9 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:24:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1567.6 tokens/s, Avg generation throughput: 439.5 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:24:24 [loggers.py:273] Engine 000: Avg prompt throughput: 2257.1 tokens/s, Avg generation throughput: 400.5 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:24:34 [loggers.py:273] Engine 000: Avg prompt throughput: 964.6 tokens/s, Avg generation throughput: 471.9 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:24:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1671.4 tokens/s, Avg generation throughput: 427.0 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:24:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1229.5 tokens/s, Avg generation throughput: 469.5 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:25:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1744.1 tokens/s, Avg generation throughput: 450.9 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:25:14 [loggers.py:273] Engine 000: Avg prompt throughput: 995.0 tokens/s, Avg generation throughput: 477.9 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:25:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1906.5 tokens/s, Avg generation throughput: 436.4 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:25:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1313.8 tokens/s, Avg generation throughput: 463.5 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:25:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1545.9 tokens/s, Avg generation throughput: 419.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:25:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1460.4 tokens/s, Avg generation throughput: 451.0 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:26:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1531.9 tokens/s, Avg generation throughput: 460.6 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:26:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1641.3 tokens/s, Avg generation throughput: 441.6 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:26:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1477.8 tokens/s, Avg generation throughput: 458.7 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:26:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1039.4 tokens/s, Avg generation throughput: 474.7 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:26:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1080.2 tokens/s, Avg generation throughput: 482.8 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:26:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1543.0 tokens/s, Avg generation throughput: 404.2 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:27:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1824.6 tokens/s, Avg generation throughput: 452.0 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:27:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1443.4 tokens/s, Avg generation throughput: 454.9 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:27:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1800.3 tokens/s, Avg generation throughput: 435.7 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:27:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1237.3 tokens/s, Avg generation throughput: 477.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:27:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1409.1 tokens/s, Avg generation throughput: 443.8 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:27:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1224.7 tokens/s, Avg generation throughput: 465.7 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:28:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2010.6 tokens/s, Avg generation throughput: 430.1 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:28:14 [loggers.py:273] Engine 000: Avg prompt throughput: 669.2 tokens/s, Avg generation throughput: 502.5 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:28:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1617.4 tokens/s, Avg generation throughput: 452.5 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:28:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1907.5 tokens/s, Avg generation throughput: 429.3 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:28:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1199.0 tokens/s, Avg generation throughput: 471.8 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:28:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1283.5 tokens/s, Avg generation throughput: 453.0 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:29:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1479.5 tokens/s, Avg generation throughput: 450.9 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:29:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1620.5 tokens/s, Avg generation throughput: 458.1 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:29:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1703.4 tokens/s, Avg generation throughput: 421.8 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:29:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1404.4 tokens/s, Avg generation throughput: 468.6 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:29:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1828.5 tokens/s, Avg generation throughput: 423.2 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:29:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1224.8 tokens/s, Avg generation throughput: 470.3 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:30:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1739.1 tokens/s, Avg generation throughput: 415.0 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:30:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1558.1 tokens/s, Avg generation throughput: 423.7 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:30:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1202.7 tokens/s, Avg generation throughput: 493.2 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:30:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1045.8 tokens/s, Avg generation throughput: 497.4 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:30:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1205.4 tokens/s, Avg generation throughput: 451.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:30:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1511.9 tokens/s, Avg generation throughput: 435.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:31:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1235.5 tokens/s, Avg generation throughput: 461.1 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:31:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1056.9 tokens/s, Avg generation throughput: 461.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:31:24 [loggers.py:273] Engine 000: Avg prompt throughput: 699.2 tokens/s, Avg generation throughput: 474.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:31:34 [loggers.py:273] Engine 000: Avg prompt throughput: 131.3 tokens/s, Avg generation throughput: 506.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:31:44 [loggers.py:273] Engine 000: Avg prompt throughput: 420.4 tokens/s, Avg generation throughput: 463.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:31:54 [loggers.py:273] Engine 000: Avg prompt throughput: 829.8 tokens/s, Avg generation throughput: 460.3 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:32:04 [loggers.py:273] Engine 000: Avg prompt throughput: 798.3 tokens/s, Avg generation throughput: 464.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:32:14 [loggers.py:273] Engine 000: Avg prompt throughput: 662.4 tokens/s, Avg generation throughput: 475.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:32:24 [loggers.py:273] Engine 000: Avg prompt throughput: 798.0 tokens/s, Avg generation throughput: 464.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:32:34 [loggers.py:273] Engine 000: Avg prompt throughput: 505.0 tokens/s, Avg generation throughput: 457.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:32:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1256.6 tokens/s, Avg generation throughput: 426.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:32:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1362.2 tokens/s, Avg generation throughput: 422.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:33:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 509.4 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:33:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1399.4 tokens/s, Avg generation throughput: 405.0 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:33:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1104.1 tokens/s, Avg generation throughput: 429.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:33:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 490.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:33:44 [loggers.py:273] Engine 000: Avg prompt throughput: 695.5 tokens/s, Avg generation throughput: 435.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:33:54 [loggers.py:273] Engine 000: Avg prompt throughput: 298.8 tokens/s, Avg generation throughput: 488.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:34:04 [loggers.py:273] Engine 000: Avg prompt throughput: 656.8 tokens/s, Avg generation throughput: 449.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:34:14 [loggers.py:273] Engine 000: Avg prompt throughput: 533.6 tokens/s, Avg generation throughput: 473.7 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:34:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1626.9 tokens/s, Avg generation throughput: 404.1 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:34:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 516.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:34:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 487.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:34:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 494.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:35:04 [loggers.py:273] Engine 000: Avg prompt throughput: 127.5 tokens/s, Avg generation throughput: 479.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:35:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1710.8 tokens/s, Avg generation throughput: 384.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:35:24 [loggers.py:273] Engine 000: Avg prompt throughput: 389.6 tokens/s, Avg generation throughput: 489.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:35:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1584.8 tokens/s, Avg generation throughput: 422.5 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:35:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1660.0 tokens/s, Avg generation throughput: 401.4 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:35:54 [loggers.py:273] Engine 000: Avg prompt throughput: 675.4 tokens/s, Avg generation throughput: 484.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:36:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1156.8 tokens/s, Avg generation throughput: 455.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:36:14 [loggers.py:273] Engine 000: Avg prompt throughput: 133.9 tokens/s, Avg generation throughput: 496.0 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:36:24 [loggers.py:273] Engine 000: Avg prompt throughput: 550.7 tokens/s, Avg generation throughput: 478.6 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:36:34 [loggers.py:273] Engine 000: Avg prompt throughput: 712.2 tokens/s, Avg generation throughput: 477.9 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:36:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1182.8 tokens/s, Avg generation throughput: 437.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:36:54 [loggers.py:273] Engine 000: Avg prompt throughput: 518.6 tokens/s, Avg generation throughput: 489.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:37:04 [loggers.py:273] Engine 000: Avg prompt throughput: 398.7 tokens/s, Avg generation throughput: 486.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:37:14 [loggers.py:273] Engine 000: Avg prompt throughput: 437.4 tokens/s, Avg generation throughput: 470.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:37:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:37:34 [loggers.py:273] Engine 000: Avg prompt throughput: 841.5 tokens/s, Avg generation throughput: 414.7 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:37:44 [loggers.py:273] Engine 000: Avg prompt throughput: 923.6 tokens/s, Avg generation throughput: 446.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:37:54 [loggers.py:273] Engine 000: Avg prompt throughput: 926.2 tokens/s, Avg generation throughput: 440.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:38:04 [loggers.py:273] Engine 000: Avg prompt throughput: 936.4 tokens/s, Avg generation throughput: 460.3 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:38:14 [loggers.py:273] Engine 000: Avg prompt throughput: 780.8 tokens/s, Avg generation throughput: 467.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:38:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1041.7 tokens/s, Avg generation throughput: 459.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:38:34 [loggers.py:273] Engine 000: Avg prompt throughput: 876.4 tokens/s, Avg generation throughput: 463.0 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:38:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1388.0 tokens/s, Avg generation throughput: 411.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:38:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1201.4 tokens/s, Avg generation throughput: 441.0 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:39:04 [loggers.py:273] Engine 000: Avg prompt throughput: 687.5 tokens/s, Avg generation throughput: 471.9 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:39:14 [loggers.py:273] Engine 000: Avg prompt throughput: 143.7 tokens/s, Avg generation throughput: 523.0 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:39:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 497.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:39:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 472.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:39:44 [loggers.py:273] Engine 000: Avg prompt throughput: 284.2 tokens/s, Avg generation throughput: 480.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:39:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1188.4 tokens/s, Avg generation throughput: 415.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:40:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1282.2 tokens/s, Avg generation throughput: 416.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:40:14 [loggers.py:273] Engine 000: Avg prompt throughput: 503.3 tokens/s, Avg generation throughput: 468.6 tokens/s, Running: 52 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:40:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1060.6 tokens/s, Avg generation throughput: 451.2 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:40:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1122.5 tokens/s, Avg generation throughput: 425.3 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:40:44 [loggers.py:273] Engine 000: Avg prompt throughput: 414.2 tokens/s, Avg generation throughput: 481.1 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:40:54 [loggers.py:273] Engine 000: Avg prompt throughput: 528.3 tokens/s, Avg generation throughput: 464.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:41:04 [loggers.py:273] Engine 000: Avg prompt throughput: 667.9 tokens/s, Avg generation throughput: 463.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:41:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1590.1 tokens/s, Avg generation throughput: 414.3 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:41:24 [loggers.py:273] Engine 000: Avg prompt throughput: 525.7 tokens/s, Avg generation throughput: 496.0 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:41:34 [loggers.py:273] Engine 000: Avg prompt throughput: 190.6 tokens/s, Avg generation throughput: 502.3 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:41:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 525.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:41:54 [loggers.py:273] Engine 000: Avg prompt throughput: 647.7 tokens/s, Avg generation throughput: 447.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:42:04 [loggers.py:273] Engine 000: Avg prompt throughput: 564.5 tokens/s, Avg generation throughput: 445.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:42:14 [loggers.py:273] Engine 000: Avg prompt throughput: 762.4 tokens/s, Avg generation throughput: 454.1 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:42:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1776.0 tokens/s, Avg generation throughput: 386.9 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:42:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1057.5 tokens/s, Avg generation throughput: 428.4 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:42:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1390.5 tokens/s, Avg generation throughput: 422.4 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:42:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1233.4 tokens/s, Avg generation throughput: 434.5 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:43:04 [loggers.py:273] Engine 000: Avg prompt throughput: 263.9 tokens/s, Avg generation throughput: 508.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:43:14 [loggers.py:273] Engine 000: Avg prompt throughput: 671.9 tokens/s, Avg generation throughput: 462.0 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:43:24 [loggers.py:273] Engine 000: Avg prompt throughput: 391.2 tokens/s, Avg generation throughput: 498.8 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:43:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1198.1 tokens/s, Avg generation throughput: 445.2 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:43:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1119.5 tokens/s, Avg generation throughput: 441.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:43:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 519.9 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:44:04 [loggers.py:273] Engine 000: Avg prompt throughput: 127.3 tokens/s, Avg generation throughput: 513.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:44:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 480.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:44:24 [loggers.py:273] Engine 000: Avg prompt throughput: 154.2 tokens/s, Avg generation throughput: 459.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:44:34 [loggers.py:273] Engine 000: Avg prompt throughput: 420.3 tokens/s, Avg generation throughput: 470.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:44:44 [loggers.py:273] Engine 000: Avg prompt throughput: 388.2 tokens/s, Avg generation throughput: 462.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:44:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1481.7 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:45:04 [loggers.py:273] Engine 000: Avg prompt throughput: 834.4 tokens/s, Avg generation throughput: 448.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:45:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1828.6 tokens/s, Avg generation throughput: 382.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:45:24 [loggers.py:273] Engine 000: Avg prompt throughput: 137.6 tokens/s, Avg generation throughput: 523.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:45:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1030.2 tokens/s, Avg generation throughput: 422.2 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:45:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1003.0 tokens/s, Avg generation throughput: 447.7 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:45:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1615.3 tokens/s, Avg generation throughput: 400.9 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:46:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1196.7 tokens/s, Avg generation throughput: 471.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:46:14 [loggers.py:273] Engine 000: Avg prompt throughput: 167.1 tokens/s, Avg generation throughput: 524.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:46:24 [loggers.py:273] Engine 000: Avg prompt throughput: 268.0 tokens/s, Avg generation throughput: 492.2 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:46:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 507.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:46:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 489.1 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:46:54 [loggers.py:273] Engine 000: Avg prompt throughput: 135.8 tokens/s, Avg generation throughput: 464.0 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:47:04 [loggers.py:273] Engine 000: Avg prompt throughput: 321.6 tokens/s, Avg generation throughput: 475.6 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:47:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1684.1 tokens/s, Avg generation throughput: 391.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:47:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1123.5 tokens/s, Avg generation throughput: 413.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:47:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1265.9 tokens/s, Avg generation throughput: 420.1 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:47:44 [loggers.py:273] Engine 000: Avg prompt throughput: 396.1 tokens/s, Avg generation throughput: 487.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:47:54 [loggers.py:273] Engine 000: Avg prompt throughput: 274.4 tokens/s, Avg generation throughput: 489.3 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:48:04 [loggers.py:273] Engine 000: Avg prompt throughput: 144.3 tokens/s, Avg generation throughput: 495.1 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:48:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1485.9 tokens/s, Avg generation throughput: 397.8 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:48:24 [loggers.py:273] Engine 000: Avg prompt throughput: 794.7 tokens/s, Avg generation throughput: 458.1 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:48:34 [loggers.py:273] Engine 000: Avg prompt throughput: 131.6 tokens/s, Avg generation throughput: 510.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:48:44 [loggers.py:273] Engine 000: Avg prompt throughput: 528.4 tokens/s, Avg generation throughput: 482.7 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:48:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 497.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:49:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 474.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:49:14 [loggers.py:273] Engine 000: Avg prompt throughput: 372.6 tokens/s, Avg generation throughput: 465.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:49:24 [loggers.py:273] Engine 000: Avg prompt throughput: 910.2 tokens/s, Avg generation throughput: 431.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:49:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1504.9 tokens/s, Avg generation throughput: 396.5 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:49:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1781.3 tokens/s, Avg generation throughput: 401.7 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:49:54 [loggers.py:273] Engine 000: Avg prompt throughput: 790.4 tokens/s, Avg generation throughput: 459.7 tokens/s, Running: 52 reqs, Waiting: 10 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:50:04 [loggers.py:273] Engine 000: Avg prompt throughput: 698.0 tokens/s, Avg generation throughput: 491.7 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:50:14 [loggers.py:273] Engine 000: Avg prompt throughput: 264.4 tokens/s, Avg generation throughput: 478.3 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:50:24 [loggers.py:273] Engine 000: Avg prompt throughput: 264.0 tokens/s, Avg generation throughput: 500.7 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:50:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1621.5 tokens/s, Avg generation throughput: 405.4 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:50:44 [loggers.py:273] Engine 000: Avg prompt throughput: 853.2 tokens/s, Avg generation throughput: 477.0 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:50:54 [loggers.py:273] Engine 000: Avg prompt throughput: 139.2 tokens/s, Avg generation throughput: 525.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:51:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 513.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:51:14 [loggers.py:273] Engine 000: Avg prompt throughput: 269.4 tokens/s, Avg generation throughput: 436.7 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:51:24 [loggers.py:273] Engine 000: Avg prompt throughput: 275.4 tokens/s, Avg generation throughput: 483.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:51:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 487.7 tokens/s, Running: 47 reqs, Waiting: 16 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:51:44 [loggers.py:273] Engine 000: Avg prompt throughput: 399.5 tokens/s, Avg generation throughput: 455.5 tokens/s, Running: 48 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:51:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1719.0 tokens/s, Avg generation throughput: 375.5 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:52:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1516.5 tokens/s, Avg generation throughput: 408.2 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:52:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1572.1 tokens/s, Avg generation throughput: 428.2 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:52:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1659.8 tokens/s, Avg generation throughput: 434.2 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:52:34 [loggers.py:273] Engine 000: Avg prompt throughput: 142.5 tokens/s, Avg generation throughput: 514.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:52:44 [loggers.py:273] Engine 000: Avg prompt throughput: 262.8 tokens/s, Avg generation throughput: 476.7 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:52:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1503.0 tokens/s, Avg generation throughput: 418.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:53:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1259.9 tokens/s, Avg generation throughput: 439.1 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:53:14 [loggers.py:273] Engine 000: Avg prompt throughput: 542.9 tokens/s, Avg generation throughput: 493.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:53:24 [loggers.py:273] Engine 000: Avg prompt throughput: 264.9 tokens/s, Avg generation throughput: 506.5 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:53:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 520.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:53:44 [loggers.py:273] Engine 000: Avg prompt throughput: 149.9 tokens/s, Avg generation throughput: 480.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:53:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 472.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:54:04 [loggers.py:273] Engine 000: Avg prompt throughput: 137.5 tokens/s, Avg generation throughput: 474.5 tokens/s, Running: 47 reqs, Waiting: 17 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:54:14 [loggers.py:273] Engine 000: Avg prompt throughput: 803.5 tokens/s, Avg generation throughput: 420.8 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:54:24 [loggers.py:273] Engine 000: Avg prompt throughput: 400.7 tokens/s, Avg generation throughput: 471.9 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:54:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1209.3 tokens/s, Avg generation throughput: 412.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:54:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2076.2 tokens/s, Avg generation throughput: 389.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:54:54 [loggers.py:273] Engine 000: Avg prompt throughput: 495.9 tokens/s, Avg generation throughput: 476.1 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:55:04 [loggers.py:273] Engine 000: Avg prompt throughput: 895.3 tokens/s, Avg generation throughput: 463.5 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:55:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1050.3 tokens/s, Avg generation throughput: 418.1 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:55:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1838.5 tokens/s, Avg generation throughput: 419.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:55:34 [loggers.py:273] Engine 000: Avg prompt throughput: 985.1 tokens/s, Avg generation throughput: 471.9 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:55:44 [loggers.py:273] Engine 000: Avg prompt throughput: 189.5 tokens/s, Avg generation throughput: 508.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:55:54 [loggers.py:273] Engine 000: Avg prompt throughput: 132.6 tokens/s, Avg generation throughput: 498.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:56:04 [loggers.py:273] Engine 000: Avg prompt throughput: 131.4 tokens/s, Avg generation throughput: 496.7 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 10:56:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 502.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:56:24 [loggers.py:273] Engine 000: Avg prompt throughput: 385.4 tokens/s, Avg generation throughput: 444.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:56:34 [loggers.py:273] Engine 000: Avg prompt throughput: 556.1 tokens/s, Avg generation throughput: 456.3 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:56:44 [loggers.py:273] Engine 000: Avg prompt throughput: 649.2 tokens/s, Avg generation throughput: 456.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:56:54 [loggers.py:273] Engine 000: Avg prompt throughput: 407.9 tokens/s, Avg generation throughput: 448.3 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:57:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1528.5 tokens/s, Avg generation throughput: 397.3 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:57:14 [loggers.py:273] Engine 000: Avg prompt throughput: 263.2 tokens/s, Avg generation throughput: 474.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:57:24 [loggers.py:273] Engine 000: Avg prompt throughput: 693.4 tokens/s, Avg generation throughput: 429.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:57:34 [loggers.py:273] Engine 000: Avg prompt throughput: 548.3 tokens/s, Avg generation throughput: 434.9 tokens/s, Running: 49 reqs, Waiting: 13 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:57:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2428.9 tokens/s, Avg generation throughput: 367.5 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:57:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1511.4 tokens/s, Avg generation throughput: 416.3 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:58:04 [loggers.py:273] Engine 000: Avg prompt throughput: 261.9 tokens/s, Avg generation throughput: 517.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:58:14 [loggers.py:273] Engine 000: Avg prompt throughput: 503.8 tokens/s, Avg generation throughput: 462.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:58:24 [loggers.py:273] Engine 000: Avg prompt throughput: 249.6 tokens/s, Avg generation throughput: 481.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:58:34 [loggers.py:273] Engine 000: Avg prompt throughput: 633.7 tokens/s, Avg generation throughput: 461.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:58:44 [loggers.py:273] Engine 000: Avg prompt throughput: 642.9 tokens/s, Avg generation throughput: 460.6 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:58:54 [loggers.py:273] Engine 000: Avg prompt throughput: 521.2 tokens/s, Avg generation throughput: 461.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:59:04 [loggers.py:273] Engine 000: Avg prompt throughput: 685.8 tokens/s, Avg generation throughput: 450.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:59:14 [loggers.py:273] Engine 000: Avg prompt throughput: 282.8 tokens/s, Avg generation throughput: 475.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:59:24 [loggers.py:273] Engine 000: Avg prompt throughput: 675.5 tokens/s, Avg generation throughput: 426.3 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:59:34 [loggers.py:273] Engine 000: Avg prompt throughput: 551.3 tokens/s, Avg generation throughput: 468.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:59:44 [loggers.py:273] Engine 000: Avg prompt throughput: 979.3 tokens/s, Avg generation throughput: 437.4 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 10:59:54 [loggers.py:273] Engine 000: Avg prompt throughput: 667.7 tokens/s, Avg generation throughput: 458.1 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:00:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1605.8 tokens/s, Avg generation throughput: 400.9 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:00:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1356.0 tokens/s, Avg generation throughput: 468.2 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:00:24 [loggers.py:273] Engine 000: Avg prompt throughput: 688.1 tokens/s, Avg generation throughput: 454.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:00:34 [loggers.py:273] Engine 000: Avg prompt throughput: 272.9 tokens/s, Avg generation throughput: 492.9 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:00:44 [loggers.py:273] Engine 000: Avg prompt throughput: 387.8 tokens/s, Avg generation throughput: 481.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:00:54 [loggers.py:273] Engine 000: Avg prompt throughput: 138.3 tokens/s, Avg generation throughput: 486.8 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:01:04 [loggers.py:273] Engine 000: Avg prompt throughput: 285.5 tokens/s, Avg generation throughput: 479.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:01:14 [loggers.py:273] Engine 000: Avg prompt throughput: 920.6 tokens/s, Avg generation throughput: 445.8 tokens/s, Running: 49 reqs, Waiting: 14 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:01:24 [loggers.py:273] Engine 000: Avg prompt throughput: 397.0 tokens/s, Avg generation throughput: 460.5 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:01:34 [loggers.py:273] Engine 000: Avg prompt throughput: 905.5 tokens/s, Avg generation throughput: 436.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:01:44 [loggers.py:273] Engine 000: Avg prompt throughput: 408.6 tokens/s, Avg generation throughput: 472.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:01:54 [loggers.py:273] Engine 000: Avg prompt throughput: 782.9 tokens/s, Avg generation throughput: 462.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:02:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1575.2 tokens/s, Avg generation throughput: 404.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:02:14 [loggers.py:273] Engine 000: Avg prompt throughput: 750.5 tokens/s, Avg generation throughput: 454.3 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:02:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1729.0 tokens/s, Avg generation throughput: 399.4 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:02:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1240.7 tokens/s, Avg generation throughput: 439.4 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:02:44 [loggers.py:273] Engine 000: Avg prompt throughput: 802.8 tokens/s, Avg generation throughput: 464.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:02:54 [loggers.py:273] Engine 000: Avg prompt throughput: 187.3 tokens/s, Avg generation throughput: 511.8 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:03:04 [loggers.py:273] Engine 000: Avg prompt throughput: 274.4 tokens/s, Avg generation throughput: 483.0 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:03:14 [loggers.py:273] Engine 000: Avg prompt throughput: 399.3 tokens/s, Avg generation throughput: 464.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:03:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 504.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:03:34 [loggers.py:273] Engine 000: Avg prompt throughput: 769.6 tokens/s, Avg generation throughput: 454.9 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:03:44 [loggers.py:273] Engine 000: Avg prompt throughput: 904.2 tokens/s, Avg generation throughput: 437.9 tokens/s, Running: 52 reqs, Waiting: 12 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:03:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1383.7 tokens/s, Avg generation throughput: 423.7 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:04:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 505.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:04:14 [loggers.py:273] Engine 000: Avg prompt throughput: 282.2 tokens/s, Avg generation throughput: 467.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:04:24 [loggers.py:273] Engine 000: Avg prompt throughput: 547.1 tokens/s, Avg generation throughput: 424.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:04:34 [loggers.py:273] Engine 000: Avg prompt throughput: 666.0 tokens/s, Avg generation throughput: 462.2 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:04:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1059.3 tokens/s, Avg generation throughput: 424.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:04:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1883.2 tokens/s, Avg generation throughput: 402.1 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:05:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1410.7 tokens/s, Avg generation throughput: 459.4 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:05:14 [loggers.py:273] Engine 000: Avg prompt throughput: 375.7 tokens/s, Avg generation throughput: 508.9 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:05:24 [loggers.py:273] Engine 000: Avg prompt throughput: 776.4 tokens/s, Avg generation throughput: 466.9 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:05:34 [loggers.py:273] Engine 000: Avg prompt throughput: 256.6 tokens/s, Avg generation throughput: 481.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:05:44 [loggers.py:273] Engine 000: Avg prompt throughput: 130.2 tokens/s, Avg generation throughput: 495.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:05:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 504.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:06:04 [loggers.py:273] Engine 000: Avg prompt throughput: 682.4 tokens/s, Avg generation throughput: 452.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:06:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1228.7 tokens/s, Avg generation throughput: 429.5 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:06:24 [loggers.py:273] Engine 000: Avg prompt throughput: 406.8 tokens/s, Avg generation throughput: 476.1 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:06:34 [loggers.py:273] Engine 000: Avg prompt throughput: 139.3 tokens/s, Avg generation throughput: 480.2 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:06:44 [loggers.py:273] Engine 000: Avg prompt throughput: 300.9 tokens/s, Avg generation throughput: 460.6 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:06:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1066.5 tokens/s, Avg generation throughput: 424.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:07:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1684.9 tokens/s, Avg generation throughput: 381.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:07:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1901.5 tokens/s, Avg generation throughput: 380.5 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:07:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1370.4 tokens/s, Avg generation throughput: 437.1 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:07:34 [loggers.py:273] Engine 000: Avg prompt throughput: 131.8 tokens/s, Avg generation throughput: 538.4 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:07:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 496.6 tokens/s, Running: 51 reqs, Waiting: 12 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:07:54 [loggers.py:273] Engine 000: Avg prompt throughput: 373.8 tokens/s, Avg generation throughput: 458.3 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:08:04 [loggers.py:273] Engine 000: Avg prompt throughput: 368.9 tokens/s, Avg generation throughput: 475.9 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:08:14 [loggers.py:273] Engine 000: Avg prompt throughput: 372.9 tokens/s, Avg generation throughput: 484.6 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:08:24 [loggers.py:273] Engine 000: Avg prompt throughput: 924.3 tokens/s, Avg generation throughput: 447.8 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:08:34 [loggers.py:273] Engine 000: Avg prompt throughput: 804.0 tokens/s, Avg generation throughput: 448.0 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:08:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1393.9 tokens/s, Avg generation throughput: 406.6 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:08:54 [loggers.py:273] Engine 000: Avg prompt throughput: 138.5 tokens/s, Avg generation throughput: 498.0 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:09:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 487.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:09:14 [loggers.py:273] Engine 000: Avg prompt throughput: 704.3 tokens/s, Avg generation throughput: 434.5 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:09:24 [loggers.py:273] Engine 000: Avg prompt throughput: 413.3 tokens/s, Avg generation throughput: 465.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:09:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2450.8 tokens/s, Avg generation throughput: 344.4 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:09:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1652.8 tokens/s, Avg generation throughput: 428.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:09:54 [loggers.py:273] Engine 000: Avg prompt throughput: 527.9 tokens/s, Avg generation throughput: 516.5 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:10:04 [loggers.py:273] Engine 000: Avg prompt throughput: 139.5 tokens/s, Avg generation throughput: 517.8 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:10:14 [loggers.py:273] Engine 000: Avg prompt throughput: 141.9 tokens/s, Avg generation throughput: 500.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:10:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 505.4 tokens/s, Running: 50 reqs, Waiting: 14 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:10:34 [loggers.py:273] Engine 000: Avg prompt throughput: 131.6 tokens/s, Avg generation throughput: 503.0 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:10:44 [loggers.py:273] Engine 000: Avg prompt throughput: 382.8 tokens/s, Avg generation throughput: 453.6 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:10:54 [loggers.py:273] Engine 000: Avg prompt throughput: 896.2 tokens/s, Avg generation throughput: 440.3 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:11:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1810.3 tokens/s, Avg generation throughput: 391.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:11:14 [loggers.py:273] Engine 000: Avg prompt throughput: 257.5 tokens/s, Avg generation throughput: 492.3 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:11:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 501.9 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:11:34 [loggers.py:273] Engine 000: Avg prompt throughput: 813.1 tokens/s, Avg generation throughput: 435.8 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:11:44 [loggers.py:273] Engine 000: Avg prompt throughput: 571.8 tokens/s, Avg generation throughput: 450.1 tokens/s, Running: 49 reqs, Waiting: 15 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:11:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2191.0 tokens/s, Avg generation throughput: 348.1 tokens/s, Running: 51 reqs, Waiting: 11 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:12:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2848.8 tokens/s, Avg generation throughput: 348.2 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:12:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2547.6 tokens/s, Avg generation throughput: 367.2 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:12:24 [loggers.py:273] Engine 000: Avg prompt throughput: 2125.8 tokens/s, Avg generation throughput: 412.1 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:12:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2298.0 tokens/s, Avg generation throughput: 378.5 tokens/s, Running: 55 reqs, Waiting: 6 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:12:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2386.9 tokens/s, Avg generation throughput: 391.4 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:12:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2070.6 tokens/s, Avg generation throughput: 378.9 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:13:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1848.7 tokens/s, Avg generation throughput: 402.9 tokens/s, Running: 53 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:13:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1786.9 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:13:24 [loggers.py:273] Engine 000: Avg prompt throughput: 1957.3 tokens/s, Avg generation throughput: 390.6 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:13:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1442.3 tokens/s, Avg generation throughput: 428.0 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:13:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2868.2 tokens/s, Avg generation throughput: 330.5 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:13:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2911.4 tokens/s, Avg generation throughput: 365.7 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:14:04 [loggers.py:273] Engine 000: Avg prompt throughput: 3385.0 tokens/s, Avg generation throughput: 332.8 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 95.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:14:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2963.0 tokens/s, Avg generation throughput: 366.1 tokens/s, Running: 54 reqs, Waiting: 7 reqs, GPU KV cache usage: 93.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:14:24 [loggers.py:273] Engine 000: Avg prompt throughput: 3012.6 tokens/s, Avg generation throughput: 355.7 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:14:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2540.3 tokens/s, Avg generation throughput: 389.2 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:14:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2286.7 tokens/s, Avg generation throughput: 408.9 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:14:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2390.9 tokens/s, Avg generation throughput: 390.8 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:15:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2432.1 tokens/s, Avg generation throughput: 402.1 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:15:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2765.0 tokens/s, Avg generation throughput: 361.6 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:15:24 [loggers.py:273] Engine 000: Avg prompt throughput: 2216.3 tokens/s, Avg generation throughput: 407.8 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:15:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2595.8 tokens/s, Avg generation throughput: 384.0 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:15:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2809.6 tokens/s, Avg generation throughput: 373.6 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:15:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2149.3 tokens/s, Avg generation throughput: 413.8 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:16:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2908.2 tokens/s, Avg generation throughput: 373.7 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:16:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2807.6 tokens/s, Avg generation throughput: 385.8 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:16:24 [loggers.py:273] Engine 000: Avg prompt throughput: 2399.8 tokens/s, Avg generation throughput: 409.3 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:16:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2696.2 tokens/s, Avg generation throughput: 375.8 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:16:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1798.7 tokens/s, Avg generation throughput: 437.2 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:16:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2369.7 tokens/s, Avg generation throughput: 388.6 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:17:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2014.4 tokens/s, Avg generation throughput: 418.2 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:17:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1872.3 tokens/s, Avg generation throughput: 429.4 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:17:24 [loggers.py:273] Engine 000: Avg prompt throughput: 2364.6 tokens/s, Avg generation throughput: 406.6 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 97.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:17:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1605.6 tokens/s, Avg generation throughput: 451.9 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:17:44 [loggers.py:273] Engine 000: Avg prompt throughput: 2473.8 tokens/s, Avg generation throughput: 384.0 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:17:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2287.1 tokens/s, Avg generation throughput: 412.4 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:18:04 [loggers.py:273] Engine 000: Avg prompt throughput: 2457.6 tokens/s, Avg generation throughput: 383.7 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36138 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:18:14 [loggers.py:273] Engine 000: Avg prompt throughput: 2160.1 tokens/s, Avg generation throughput: 412.1 tokens/s, Running: 55 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36308 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36272 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36224 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36258 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36358 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36066 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36336 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36184 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36312 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36214 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:18:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 440.0 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36298 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36226 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:18:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 305.1 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36002 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36370 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36286 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:18:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 238.2 tokens/s, Running: 20 reqs, Waiting: 0 reqs, GPU KV cache usage: 39.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36250 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36346 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:18:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 201.7 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36288 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:19:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 149.4 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36242 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:19:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 119.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:19:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 98.1 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:19:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 80.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:19:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 44.9 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36324 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:19:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36294 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:20:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1421.5 tokens/s, Avg generation throughput: 193.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:20:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 652.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:20:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 640.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:20:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 639.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:20:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 633.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:20:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 627.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:21:04 [loggers.py:273] Engine 000: Avg prompt throughput: 38.1 tokens/s, Avg generation throughput: 614.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:21:14 [loggers.py:273] Engine 000: Avg prompt throughput: 37.1 tokens/s, Avg generation throughput: 620.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:21:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 620.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:21:34 [loggers.py:273] Engine 000: Avg prompt throughput: 18.6 tokens/s, Avg generation throughput: 614.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:21:44 [loggers.py:273] Engine 000: Avg prompt throughput: 15.4 tokens/s, Avg generation throughput: 603.4 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:21:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 594.7 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:22:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 585.5 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37606 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37646 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37728 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37888 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38014 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38050 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38088 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38140 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:22:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1236.2 tokens/s, Avg generation throughput: 485.7 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:22:24 [loggers.py:273] Engine 000: Avg prompt throughput: 498.0 tokens/s, Avg generation throughput: 626.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:22:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 639.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:22:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 620.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:22:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 627.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:23:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 627.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:23:14 [loggers.py:273] Engine 000: Avg prompt throughput: 37.8 tokens/s, Avg generation throughput: 614.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:23:24 [loggers.py:273] Engine 000: Avg prompt throughput: 87.7 tokens/s, Avg generation throughput: 613.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:23:34 [loggers.py:273] Engine 000: Avg prompt throughput: 12.5 tokens/s, Avg generation throughput: 620.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37606 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:23:44 [loggers.py:273] Engine 000: Avg prompt throughput: 73.1 tokens/s, Avg generation throughput: 607.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:23:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 611.4 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:24:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 598.7 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:24:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 570.1 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:24:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 581.6 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37646 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37728 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37888 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38014 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38050 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38140 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38088 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:24:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1273.1 tokens/s, Avg generation throughput: 535.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:24:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 639.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:24:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 633.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:25:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 633.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:25:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 620.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:25:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 620.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:25:34 [loggers.py:273] Engine 000: Avg prompt throughput: 71.6 tokens/s, Avg generation throughput: 600.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37738 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:25:44 [loggers.py:273] Engine 000: Avg prompt throughput: 43.8 tokens/s, Avg generation throughput: 596.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:25:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 584.7 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:26:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 582.0 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:26:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 580.8 tokens/s, Running: 59 reqs, Waiting: 0 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:26:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 566.0 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37606 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:26:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 558.5 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:38008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37786 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38160 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37628 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37646 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37728 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37772 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37862 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37952 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37878 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38014 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38050 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37888 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38140 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38088 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38076 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:38124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:26:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 285.5 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:26:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 65.4 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:27:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 66.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:38036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:27:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 57.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:27:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 54.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:27:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.2 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:38106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:37918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:27:44 [loggers.py:273] Engine 000: Avg prompt throughput: 3058.9 tokens/s, Avg generation throughput: 42.2 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 58.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:27:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1897.9 tokens/s, Avg generation throughput: 514.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:28:04 [loggers.py:273] Engine 000: Avg prompt throughput: 77.4 tokens/s, Avg generation throughput: 627.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:28:14 [loggers.py:273] Engine 000: Avg prompt throughput: 598.5 tokens/s, Avg generation throughput: 567.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 90.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:28:24 [loggers.py:273] Engine 000: Avg prompt throughput: 451.6 tokens/s, Avg generation throughput: 587.3 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:28:34 [loggers.py:273] Engine 000: Avg prompt throughput: 224.4 tokens/s, Avg generation throughput: 591.9 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:28:44 [loggers.py:273] Engine 000: Avg prompt throughput: 534.2 tokens/s, Avg generation throughput: 571.9 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:28:54 [loggers.py:273] Engine 000: Avg prompt throughput: 148.6 tokens/s, Avg generation throughput: 585.2 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:29:04 [loggers.py:273] Engine 000: Avg prompt throughput: 332.7 tokens/s, Avg generation throughput: 552.9 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:29:14 [loggers.py:273] Engine 000: Avg prompt throughput: 78.8 tokens/s, Avg generation throughput: 567.6 tokens/s, Running: 60 reqs, Waiting: 4 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:29:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 550.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:29:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 512.6 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:29:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 533.4 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:29:54 [loggers.py:273] Engine 000: Avg prompt throughput: 81.7 tokens/s, Avg generation throughput: 493.0 tokens/s, Running: 53 reqs, Waiting: 11 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:30:04 [loggers.py:273] Engine 000: Avg prompt throughput: 1742.3 tokens/s, Avg generation throughput: 399.4 tokens/s, Running: 59 reqs, Waiting: 3 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:30:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1934.7 tokens/s, Avg generation throughput: 481.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 95.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:30:24 [loggers.py:273] Engine 000: Avg prompt throughput: 152.5 tokens/s, Avg generation throughput: 601.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:30:34 [loggers.py:273] Engine 000: Avg prompt throughput: 560.2 tokens/s, Avg generation throughput: 561.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 95.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:30:44 [loggers.py:273] Engine 000: Avg prompt throughput: 558.3 tokens/s, Avg generation throughput: 568.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:30:54 [loggers.py:273] Engine 000: Avg prompt throughput: 220.9 tokens/s, Avg generation throughput: 565.6 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:31:04 [loggers.py:273] Engine 000: Avg prompt throughput: 614.7 tokens/s, Avg generation throughput: 557.2 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:31:14 [loggers.py:273] Engine 000: Avg prompt throughput: 151.0 tokens/s, Avg generation throughput: 585.6 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:31:24 [loggers.py:273] Engine 000: Avg prompt throughput: 73.9 tokens/s, Avg generation throughput: 570.4 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:31:34 [loggers.py:273] Engine 000: Avg prompt throughput: 225.5 tokens/s, Avg generation throughput: 558.6 tokens/s, Running: 59 reqs, Waiting: 4 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:31:44 [loggers.py:273] Engine 000: Avg prompt throughput: 73.1 tokens/s, Avg generation throughput: 556.7 tokens/s, Running: 55 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:31:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 515.7 tokens/s, Running: 54 reqs, Waiting: 9 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:32:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 516.2 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:32:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 512.8 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:32:24 [loggers.py:273] Engine 000: Avg prompt throughput: 554.3 tokens/s, Avg generation throughput: 463.4 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 96.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:32:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2732.1 tokens/s, Avg generation throughput: 418.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:32:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 614.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:32:54 [loggers.py:273] Engine 000: Avg prompt throughput: 1154.5 tokens/s, Avg generation throughput: 509.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:33:04 [loggers.py:273] Engine 000: Avg prompt throughput: 711.5 tokens/s, Avg generation throughput: 555.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:33:14 [loggers.py:273] Engine 000: Avg prompt throughput: 227.3 tokens/s, Avg generation throughput: 588.6 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:33:24 [loggers.py:273] Engine 000: Avg prompt throughput: 462.0 tokens/s, Avg generation throughput: 558.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:33:34 [loggers.py:273] Engine 000: Avg prompt throughput: 155.2 tokens/s, Avg generation throughput: 563.2 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:33:44 [loggers.py:273] Engine 000: Avg prompt throughput: 150.1 tokens/s, Avg generation throughput: 584.9 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:33:54 [loggers.py:273] Engine 000: Avg prompt throughput: 150.0 tokens/s, Avg generation throughput: 573.0 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:34:04 [loggers.py:273] Engine 000: Avg prompt throughput: 151.4 tokens/s, Avg generation throughput: 565.1 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:34:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 516.5 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:34:24 [loggers.py:273] Engine 000: Avg prompt throughput: 72.8 tokens/s, Avg generation throughput: 531.8 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:34:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 526.9 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:34:44 [loggers.py:273] Engine 000: Avg prompt throughput: 75.2 tokens/s, Avg generation throughput: 510.0 tokens/s, Running: 54 reqs, Waiting: 10 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:34:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2526.8 tokens/s, Avg generation throughput: 396.1 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:35:04 [loggers.py:273] Engine 000: Avg prompt throughput: 75.7 tokens/s, Avg generation throughput: 576.8 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:35:14 [loggers.py:273] Engine 000: Avg prompt throughput: 713.8 tokens/s, Avg generation throughput: 517.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:35:24 [loggers.py:273] Engine 000: Avg prompt throughput: 763.7 tokens/s, Avg generation throughput: 542.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:35:34 [loggers.py:273] Engine 000: Avg prompt throughput: 376.2 tokens/s, Avg generation throughput: 558.6 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:35:44 [loggers.py:273] Engine 000: Avg prompt throughput: 868.2 tokens/s, Avg generation throughput: 532.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:35:54 [loggers.py:273] Engine 000: Avg prompt throughput: 302.2 tokens/s, Avg generation throughput: 579.3 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:36:04 [loggers.py:273] Engine 000: Avg prompt throughput: 524.3 tokens/s, Avg generation throughput: 549.0 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:36:14 [loggers.py:273] Engine 000: Avg prompt throughput: 303.9 tokens/s, Avg generation throughput: 578.8 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:36:24 [loggers.py:273] Engine 000: Avg prompt throughput: 300.9 tokens/s, Avg generation throughput: 552.6 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:36:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 572.0 tokens/s, Running: 58 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:36:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 547.8 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:36:54 [loggers.py:273] Engine 000: Avg prompt throughput: 223.5 tokens/s, Avg generation throughput: 509.5 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:37:04 [loggers.py:273] Engine 000: Avg prompt throughput: 318.9 tokens/s, Avg generation throughput: 511.6 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:37:14 [loggers.py:273] Engine 000: Avg prompt throughput: 1400.4 tokens/s, Avg generation throughput: 447.0 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:37:24 [loggers.py:273] Engine 000: Avg prompt throughput: 882.8 tokens/s, Avg generation throughput: 534.6 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:37:34 [loggers.py:273] Engine 000: Avg prompt throughput: 299.9 tokens/s, Avg generation throughput: 573.9 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:37:44 [loggers.py:273] Engine 000: Avg prompt throughput: 654.6 tokens/s, Avg generation throughput: 543.0 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:37:54 [loggers.py:273] Engine 000: Avg prompt throughput: 475.6 tokens/s, Avg generation throughput: 538.5 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:38:04 [loggers.py:273] Engine 000: Avg prompt throughput: 676.7 tokens/s, Avg generation throughput: 522.9 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:38:14 [loggers.py:273] Engine 000: Avg prompt throughput: 453.3 tokens/s, Avg generation throughput: 563.8 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:38:24 [loggers.py:273] Engine 000: Avg prompt throughput: 543.6 tokens/s, Avg generation throughput: 554.2 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:38:34 [loggers.py:273] Engine 000: Avg prompt throughput: 469.9 tokens/s, Avg generation throughput: 567.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:38:44 [loggers.py:273] Engine 000: Avg prompt throughput: 373.0 tokens/s, Avg generation throughput: 568.1 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:38:54 [loggers.py:273] Engine 000: Avg prompt throughput: 223.9 tokens/s, Avg generation throughput: 544.5 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:39:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 560.3 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:39:14 [loggers.py:273] Engine 000: Avg prompt throughput: 81.0 tokens/s, Avg generation throughput: 541.1 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:39:24 [loggers.py:273] Engine 000: Avg prompt throughput: 220.0 tokens/s, Avg generation throughput: 519.1 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:39:34 [loggers.py:273] Engine 000: Avg prompt throughput: 486.2 tokens/s, Avg generation throughput: 513.6 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:39:44 [loggers.py:273] Engine 000: Avg prompt throughput: 1546.2 tokens/s, Avg generation throughput: 473.7 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:39:54 [loggers.py:273] Engine 000: Avg prompt throughput: 237.2 tokens/s, Avg generation throughput: 576.3 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:40:04 [loggers.py:273] Engine 000: Avg prompt throughput: 791.7 tokens/s, Avg generation throughput: 524.0 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:40:14 [loggers.py:273] Engine 000: Avg prompt throughput: 384.8 tokens/s, Avg generation throughput: 548.8 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 99.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:40:24 [loggers.py:273] Engine 000: Avg prompt throughput: 317.6 tokens/s, Avg generation throughput: 550.6 tokens/s, Running: 61 reqs, Waiting: 3 reqs, GPU KV cache usage: 100.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:40:34 [loggers.py:273] Engine 000: Avg prompt throughput: 627.3 tokens/s, Avg generation throughput: 536.9 tokens/s, Running: 62 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48696 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48750 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:40:44 [loggers.py:273] Engine 000: Avg prompt throughput: 229.5 tokens/s, Avg generation throughput: 558.7 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49192 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:40:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 516.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49164 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:41:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 454.3 tokens/s, Running: 40 reqs, Waiting: 0 reqs, GPU KV cache usage: 66.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:41:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 414.4 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:41:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 403.2 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 67.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49096 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:41:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 386.6 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:41:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 378.7 tokens/s, Running: 35 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49222 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48670 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49104 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:41:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 310.1 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 11:42:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 239.8 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49094 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:42:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 218.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 30.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:49020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:42:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 140.8 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49080 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49026 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:42:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 99.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:49006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35614 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:42:44 [loggers.py:273] Engine 000: Avg prompt throughput: 366.7 tokens/s, Avg generation throughput: 105.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:42:54 [loggers.py:273] Engine 000: Avg prompt throughput: 107.7 tokens/s, Avg generation throughput: 598.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35614 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:43:04 [loggers.py:273] Engine 000: Avg prompt throughput: 103.1 tokens/s, Avg generation throughput: 591.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:43:14 [loggers.py:273] Engine 000: Avg prompt throughput: 82.7 tokens/s, Avg generation throughput: 586.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:43:24 [loggers.py:273] Engine 000: Avg prompt throughput: 109.5 tokens/s, Avg generation throughput: 584.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:43:34 [loggers.py:273] Engine 000: Avg prompt throughput: 80.3 tokens/s, Avg generation throughput: 592.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:43:44 [loggers.py:273] Engine 000: Avg prompt throughput: 62.0 tokens/s, Avg generation throughput: 586.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:43:54 [loggers.py:273] Engine 000: Avg prompt throughput: 84.9 tokens/s, Avg generation throughput: 592.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:44:04 [loggers.py:273] Engine 000: Avg prompt throughput: 61.6 tokens/s, Avg generation throughput: 593.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:44:14 [loggers.py:273] Engine 000: Avg prompt throughput: 110.0 tokens/s, Avg generation throughput: 591.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:44:24 [loggers.py:273] Engine 000: Avg prompt throughput: 80.3 tokens/s, Avg generation throughput: 585.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:44:34 [loggers.py:273] Engine 000: Avg prompt throughput: 106.9 tokens/s, Avg generation throughput: 585.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:44:44 [loggers.py:273] Engine 000: Avg prompt throughput: 84.4 tokens/s, Avg generation throughput: 585.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:44:54 [loggers.py:273] Engine 000: Avg prompt throughput: 81.2 tokens/s, Avg generation throughput: 585.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:45:04 [loggers.py:273] Engine 000: Avg prompt throughput: 100.9 tokens/s, Avg generation throughput: 584.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:45:14 [loggers.py:273] Engine 000: Avg prompt throughput: 136.1 tokens/s, Avg generation throughput: 584.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35614 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:45:24 [loggers.py:273] Engine 000: Avg prompt throughput: 137.2 tokens/s, Avg generation throughput: 577.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:45:34 [loggers.py:273] Engine 000: Avg prompt throughput: 101.6 tokens/s, Avg generation throughput: 591.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:45:44 [loggers.py:273] Engine 000: Avg prompt throughput: 189.9 tokens/s, Avg generation throughput: 563.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:45:54 [loggers.py:273] Engine 000: Avg prompt throughput: 77.1 tokens/s, Avg generation throughput: 592.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:46:04 [loggers.py:273] Engine 000: Avg prompt throughput: 72.8 tokens/s, Avg generation throughput: 592.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:46:14 [loggers.py:273] Engine 000: Avg prompt throughput: 110.7 tokens/s, Avg generation throughput: 578.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:46:24 [loggers.py:273] Engine 000: Avg prompt throughput: 105.9 tokens/s, Avg generation throughput: 585.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:46:34 [loggers.py:273] Engine 000: Avg prompt throughput: 53.2 tokens/s, Avg generation throughput: 586.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:46:44 [loggers.py:273] Engine 000: Avg prompt throughput: 120.6 tokens/s, Avg generation throughput: 579.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:46:54 [loggers.py:273] Engine 000: Avg prompt throughput: 77.3 tokens/s, Avg generation throughput: 585.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:47:04 [loggers.py:273] Engine 000: Avg prompt throughput: 124.5 tokens/s, Avg generation throughput: 578.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:47:14 [loggers.py:273] Engine 000: Avg prompt throughput: 80.3 tokens/s, Avg generation throughput: 579.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:47:24 [loggers.py:273] Engine 000: Avg prompt throughput: 68.8 tokens/s, Avg generation throughput: 586.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36010 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35936 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:47:34 [loggers.py:273] Engine 000: Avg prompt throughput: 95.6 tokens/s, Avg generation throughput: 569.0 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35614 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:47:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 500.8 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 64.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:47:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 479.9 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 59.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35812 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35678 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:48:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 429.4 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35830 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:48:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 399.2 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 51.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36110 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36126 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:48:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 349.6 tokens/s, Running: 29 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:36004 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35654 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35986 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:48:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 273.2 tokens/s, Running: 22 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35610 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36032 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36158 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:48:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 215.7 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35658 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35852 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:48:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 161.0 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35866 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:49:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 128.1 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:49:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 82.6 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35744 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:36124 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:35906 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:49:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 28.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:35588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:49:34 [loggers.py:273] Engine 000: Avg prompt throughput: 534.4 tokens/s, Avg generation throughput: 225.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:49:44 [loggers.py:273] Engine 000: Avg prompt throughput: 18.7 tokens/s, Avg generation throughput: 665.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:49:54 [loggers.py:273] Engine 000: Avg prompt throughput: 106.1 tokens/s, Avg generation throughput: 630.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:50:04 [loggers.py:273] Engine 000: Avg prompt throughput: 125.5 tokens/s, Avg generation throughput: 617.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:50:14 [loggers.py:273] Engine 000: Avg prompt throughput: 213.2 tokens/s, Avg generation throughput: 596.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:50:24 [loggers.py:273] Engine 000: Avg prompt throughput: 136.2 tokens/s, Avg generation throughput: 623.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:50:34 [loggers.py:273] Engine 000: Avg prompt throughput: 118.4 tokens/s, Avg generation throughput: 618.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:50:44 [loggers.py:273] Engine 000: Avg prompt throughput: 175.0 tokens/s, Avg generation throughput: 610.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:50:54 [loggers.py:273] Engine 000: Avg prompt throughput: 117.4 tokens/s, Avg generation throughput: 617.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:51:04 [loggers.py:273] Engine 000: Avg prompt throughput: 119.8 tokens/s, Avg generation throughput: 618.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:51:14 [loggers.py:273] Engine 000: Avg prompt throughput: 149.9 tokens/s, Avg generation throughput: 617.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:51:24 [loggers.py:273] Engine 000: Avg prompt throughput: 108.0 tokens/s, Avg generation throughput: 624.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:51:34 [loggers.py:273] Engine 000: Avg prompt throughput: 171.5 tokens/s, Avg generation throughput: 610.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:51:44 [loggers.py:273] Engine 000: Avg prompt throughput: 97.8 tokens/s, Avg generation throughput: 617.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:51:54 [loggers.py:273] Engine 000: Avg prompt throughput: 145.0 tokens/s, Avg generation throughput: 623.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:52:04 [loggers.py:273] Engine 000: Avg prompt throughput: 133.8 tokens/s, Avg generation throughput: 617.2 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:52:14 [loggers.py:273] Engine 000: Avg prompt throughput: 138.4 tokens/s, Avg generation throughput: 617.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:52:24 [loggers.py:273] Engine 000: Avg prompt throughput: 119.6 tokens/s, Avg generation throughput: 617.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:52:34 [loggers.py:273] Engine 000: Avg prompt throughput: 151.6 tokens/s, Avg generation throughput: 617.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:52:44 [loggers.py:273] Engine 000: Avg prompt throughput: 156.3 tokens/s, Avg generation throughput: 604.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:52:54 [loggers.py:273] Engine 000: Avg prompt throughput: 127.4 tokens/s, Avg generation throughput: 624.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:53:04 [loggers.py:273] Engine 000: Avg prompt throughput: 127.5 tokens/s, Avg generation throughput: 617.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:53:14 [loggers.py:273] Engine 000: Avg prompt throughput: 168.6 tokens/s, Avg generation throughput: 610.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:53:24 [loggers.py:273] Engine 000: Avg prompt throughput: 127.0 tokens/s, Avg generation throughput: 618.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:53:34 [loggers.py:273] Engine 000: Avg prompt throughput: 158.5 tokens/s, Avg generation throughput: 617.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:53:44 [loggers.py:273] Engine 000: Avg prompt throughput: 134.9 tokens/s, Avg generation throughput: 617.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:53:54 [loggers.py:273] Engine 000: Avg prompt throughput: 143.7 tokens/s, Avg generation throughput: 617.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:54:04 [loggers.py:273] Engine 000: Avg prompt throughput: 148.7 tokens/s, Avg generation throughput: 610.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:54:14 [loggers.py:273] Engine 000: Avg prompt throughput: 105.0 tokens/s, Avg generation throughput: 618.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:54:24 [loggers.py:273] Engine 000: Avg prompt throughput: 192.6 tokens/s, Avg generation throughput: 603.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:54:34 [loggers.py:273] Engine 000: Avg prompt throughput: 137.5 tokens/s, Avg generation throughput: 617.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:54:44 [loggers.py:273] Engine 000: Avg prompt throughput: 97.3 tokens/s, Avg generation throughput: 624.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:54:54 [loggers.py:273] Engine 000: Avg prompt throughput: 150.7 tokens/s, Avg generation throughput: 617.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:55:04 [loggers.py:273] Engine 000: Avg prompt throughput: 212.1 tokens/s, Avg generation throughput: 602.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:55:14 [loggers.py:273] Engine 000: Avg prompt throughput: 137.5 tokens/s, Avg generation throughput: 617.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:55:24 [loggers.py:273] Engine 000: Avg prompt throughput: 70.2 tokens/s, Avg generation throughput: 625.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:55:34 [loggers.py:273] Engine 000: Avg prompt throughput: 192.1 tokens/s, Avg generation throughput: 603.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:55:44 [loggers.py:273] Engine 000: Avg prompt throughput: 117.5 tokens/s, Avg generation throughput: 624.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:55:54 [loggers.py:273] Engine 000: Avg prompt throughput: 172.7 tokens/s, Avg generation throughput: 616.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:56:04 [loggers.py:273] Engine 000: Avg prompt throughput: 131.2 tokens/s, Avg generation throughput: 617.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:56:14 [loggers.py:273] Engine 000: Avg prompt throughput: 138.3 tokens/s, Avg generation throughput: 623.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:56:24 [loggers.py:273] Engine 000: Avg prompt throughput: 196.3 tokens/s, Avg generation throughput: 603.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:56:34 [loggers.py:273] Engine 000: Avg prompt throughput: 124.6 tokens/s, Avg generation throughput: 624.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:56:44 [loggers.py:273] Engine 000: Avg prompt throughput: 115.0 tokens/s, Avg generation throughput: 624.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:56:54 [loggers.py:273] Engine 000: Avg prompt throughput: 180.6 tokens/s, Avg generation throughput: 610.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:57:04 [loggers.py:273] Engine 000: Avg prompt throughput: 155.1 tokens/s, Avg generation throughput: 617.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:57:14 [loggers.py:273] Engine 000: Avg prompt throughput: 121.1 tokens/s, Avg generation throughput: 624.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:57:24 [loggers.py:273] Engine 000: Avg prompt throughput: 196.1 tokens/s, Avg generation throughput: 603.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:57:34 [loggers.py:273] Engine 000: Avg prompt throughput: 107.8 tokens/s, Avg generation throughput: 624.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:57:44 [loggers.py:273] Engine 000: Avg prompt throughput: 112.3 tokens/s, Avg generation throughput: 624.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:57:54 [loggers.py:273] Engine 000: Avg prompt throughput: 132.8 tokens/s, Avg generation throughput: 617.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:58:04 [loggers.py:273] Engine 000: Avg prompt throughput: 76.3 tokens/s, Avg generation throughput: 631.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:58:14 [loggers.py:273] Engine 000: Avg prompt throughput: 153.4 tokens/s, Avg generation throughput: 610.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:58:24 [loggers.py:273] Engine 000: Avg prompt throughput: 193.6 tokens/s, Avg generation throughput: 616.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:58:34 [loggers.py:273] Engine 000: Avg prompt throughput: 228.7 tokens/s, Avg generation throughput: 602.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:58:44 [loggers.py:273] Engine 000: Avg prompt throughput: 95.6 tokens/s, Avg generation throughput: 625.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:58:54 [loggers.py:273] Engine 000: Avg prompt throughput: 83.7 tokens/s, Avg generation throughput: 631.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:59:04 [loggers.py:273] Engine 000: Avg prompt throughput: 194.9 tokens/s, Avg generation throughput: 603.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:59:14 [loggers.py:273] Engine 000: Avg prompt throughput: 131.7 tokens/s, Avg generation throughput: 618.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:59:24 [loggers.py:273] Engine 000: Avg prompt throughput: 143.5 tokens/s, Avg generation throughput: 617.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:59:34 [loggers.py:273] Engine 000: Avg prompt throughput: 108.8 tokens/s, Avg generation throughput: 624.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:59:44 [loggers.py:273] Engine 000: Avg prompt throughput: 142.2 tokens/s, Avg generation throughput: 610.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 11:59:54 [loggers.py:273] Engine 000: Avg prompt throughput: 141.7 tokens/s, Avg generation throughput: 624.1 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:00:04 [loggers.py:273] Engine 000: Avg prompt throughput: 118.6 tokens/s, Avg generation throughput: 618.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:00:14 [loggers.py:273] Engine 000: Avg prompt throughput: 127.7 tokens/s, Avg generation throughput: 617.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:00:24 [loggers.py:273] Engine 000: Avg prompt throughput: 114.9 tokens/s, Avg generation throughput: 618.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:00:34 [loggers.py:273] Engine 000: Avg prompt throughput: 149.4 tokens/s, Avg generation throughput: 604.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:00:44 [loggers.py:273] Engine 000: Avg prompt throughput: 140.0 tokens/s, Avg generation throughput: 617.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:00:54 [loggers.py:273] Engine 000: Avg prompt throughput: 184.7 tokens/s, Avg generation throughput: 610.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:01:04 [loggers.py:273] Engine 000: Avg prompt throughput: 115.6 tokens/s, Avg generation throughput: 617.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:01:14 [loggers.py:273] Engine 000: Avg prompt throughput: 148.0 tokens/s, Avg generation throughput: 624.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:01:24 [loggers.py:273] Engine 000: Avg prompt throughput: 138.9 tokens/s, Avg generation throughput: 611.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:01:34 [loggers.py:273] Engine 000: Avg prompt throughput: 135.9 tokens/s, Avg generation throughput: 630.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:01:44 [loggers.py:273] Engine 000: Avg prompt throughput: 150.8 tokens/s, Avg generation throughput: 617.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:01:54 [loggers.py:273] Engine 000: Avg prompt throughput: 131.8 tokens/s, Avg generation throughput: 617.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:02:04 [loggers.py:273] Engine 000: Avg prompt throughput: 179.6 tokens/s, Avg generation throughput: 610.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:02:14 [loggers.py:273] Engine 000: Avg prompt throughput: 134.7 tokens/s, Avg generation throughput: 624.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47622 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47712 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47798 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:02:24 [loggers.py:273] Engine 000: Avg prompt throughput: 70.8 tokens/s, Avg generation throughput: 584.3 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 56.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48030 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47608 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47684 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47600 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47680 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47508 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:02:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 395.3 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47822 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47722 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:02:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 207.2 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47638 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47666 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47510 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:02:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 107.9 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48118 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47672 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:48116 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:03:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 53.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:03:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:03:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:03:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:03:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:03:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:55604 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1215265) INFO: 127.0.0.1:55604 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1215265) INFO: 127.0.0.1:55604 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1215265) INFO: 127.0.0.1:47548 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47732 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47806 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47856 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47394 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47710 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:04:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1385.9 tokens/s, Avg generation throughput: 88.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 68.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47732 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47824 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47838 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47380 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47380 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:04:44 [loggers.py:273] Engine 000: Avg prompt throughput: 249.1 tokens/s, Avg generation throughput: 604.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47380 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:04:54 [loggers.py:273] Engine 000: Avg prompt throughput: 127.8 tokens/s, Avg generation throughput: 618.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47426 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47460 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47460 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47458 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47856 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:05:04 [loggers.py:273] Engine 000: Avg prompt throughput: 152.3 tokens/s, Avg generation throughput: 611.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 73.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47460 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47856 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47460 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47856 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:05:14 [loggers.py:273] Engine 000: Avg prompt throughput: 56.7 tokens/s, Avg generation throughput: 638.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:05:24 [loggers.py:273] Engine 000: Avg prompt throughput: 29.7 tokens/s, Avg generation throughput: 652.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:05:34 [loggers.py:273] Engine 000: Avg prompt throughput: 20.6 tokens/s, Avg generation throughput: 652.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:05:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 652.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:05:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 659.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:06:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 659.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 93.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:06:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 654.0 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 99.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47374 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47410 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47434 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47484 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47498 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47536 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47606 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47628 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47654 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47660 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47676 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47686 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47702 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47718 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47744 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47762 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47848 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47862 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47548 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47806 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47472 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47642 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47744 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47734 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47862 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47394 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47604 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47710 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47566 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47526 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47788 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47486 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47676 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47606 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47654 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47686 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47498 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47732 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47718 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47810 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47552 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47824 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47530 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47702 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47838 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47592 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47698 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:06:24 [loggers.py:273] Engine 000: Avg prompt throughput: 304.6 tokens/s, Avg generation throughput: 394.6 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47532 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47380 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47596 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:06:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 272.0 tokens/s, Running: 23 reqs, Waiting: 0 reqs, GPU KV cache usage: 27.5%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47426 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47736 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47458 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47582 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:06:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 236.2 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.3%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47720 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47460 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47856 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:06:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 192.5 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:07:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 176.0 tokens/s, Running: 16 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47696 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:07:14 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 172.0 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:07:24 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 165.0 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:07:34 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 163.5 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:07:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 165.0 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:47374 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47410 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47414 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47434 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47484 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47536 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47558 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47628 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47660 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47776 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47786 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:47762 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:07:54 [loggers.py:273] Engine 000: Avg prompt throughput: 2817.0 tokens/s, Avg generation throughput: 105.6 tokens/s, Running: 31 reqs, Waiting: 33 reqs, GPU KV cache usage: 44.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:57950 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57946 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60756 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60858 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57978 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57990 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58052 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32894 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57972 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:08:04 [loggers.py:273] Engine 000: Avg prompt throughput: 4734.2 tokens/s, Avg generation throughput: 258.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.4%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32770 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60756 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58094 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32888 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58140 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57950 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60908 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57990 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60854 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57946 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:08:14 [loggers.py:273] Engine 000: Avg prompt throughput: 4179.1 tokens/s, Avg generation throughput: 294.6 tokens/s, Running: 62 reqs, Waiting: 2 reqs, GPU KV cache usage: 87.1%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:58124 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32770 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57972 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60854 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57950 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57990 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60908 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57978 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58124 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:08:24 [loggers.py:273] Engine 000: Avg prompt throughput: 3493.4 tokens/s, Avg generation throughput: 355.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 89.8%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58094 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57978 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32888 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60756 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60908 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57972 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57950 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58124 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:08:34 [loggers.py:273] Engine 000: Avg prompt throughput: 2905.1 tokens/s, Avg generation throughput: 381.3 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 85.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58094 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57972 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60756 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32888 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58124 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:08:44 [loggers.py:273] Engine 000: Avg prompt throughput: 3181.4 tokens/s, Avg generation throughput: 388.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:60980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60858 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57950 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58052 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32894 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60858 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58140 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:08:54 [loggers.py:273] Engine 000: Avg prompt throughput: 3094.2 tokens/s, Avg generation throughput: 374.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:32872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57946 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58124 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60858 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32894 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32770 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57946 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60854 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57990 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60858 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:09:04 [loggers.py:273] Engine 000: Avg prompt throughput: 3062.7 tokens/s, Avg generation throughput: 387.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:60854 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57946 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32894 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32770 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57978 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:09:14 [loggers.py:273] Engine 000: Avg prompt throughput: 3234.3 tokens/s, Avg generation throughput: 380.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 87.9%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60908 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57946 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32770 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58094 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57972 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57978 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58094 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60756 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60908 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:09:24 [loggers.py:273] Engine 000: Avg prompt throughput: 3202.1 tokens/s, Avg generation throughput: 373.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.2%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57978 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32888 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60908 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57972 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60794 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60996 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60754 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58006 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58044 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60864 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60956 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58078 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60988 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32770 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32826 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32842 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60940 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57972 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58030 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58094 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57950 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32850 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58052 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58046 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57954 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60748 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57934 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32782 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60774 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58108 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32796 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58140 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58110 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:09:34 [loggers.py:273] Engine 000: Avg prompt throughput: 1685.5 tokens/s, Avg generation throughput: 389.9 tokens/s, Running: 34 reqs, Waiting: 0 reqs, GPU KV cache usage: 46.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:60836 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58124 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58022 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60764 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60908 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60980 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32872 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60896 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57990 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60858 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60854 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32894 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32812 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60926 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:09:44 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 257.0 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.7%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO: 127.0.0.1:60790 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60968 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57946 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60910 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60860 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60808 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58066 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60756 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60840 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58114 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57928 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60822 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57978 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:32888 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:58014 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60886 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:60912 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO: 127.0.0.1:57962 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1215265) INFO 07-24 12:09:54 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 93.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:10:04 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1215265) INFO 07-24 12:12:45 [launcher.py:100] [shutdown] API server: shutdown triggered (APIServer pid=1215265) INFO 07-24 12:12:45 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s (APIServer pid=1215265) INFO 07-24 12:12:45 [core_client.py:655] [shutdown] MPClient: start timeout=0s (APIServer pid=1215265) INFO 07-24 12:12:45 [core_client.py:657] [shutdown] MPClient: stopping engine manager (APIServer pid=1215265) WARNING 07-24 12:12:45 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 (EngineCore pid=1215449) INFO 07-24 12:12:45 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=1215449) INFO 07-24 12:12:45 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=1215449) INFO 07-24 12:12:45 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=1215449) INFO 07-24 12:12:45 [core.py:1227] [shutdown] EngineCore: exiting busy loop (EngineCore pid=1215449) INFO 07-24 12:12:45 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2 (Worker_PP0 pid=1215563) INFO 07-24 12:12:45 [multiproc_executor.py:790] Parent process exited, terminating worker queues (APIServer pid=1215265) INFO 07-24 12:12:45 [core_client.py:659] [shutdown] MPClient: engine manager stopped (APIServer pid=1215265) INFO 07-24 12:12:45 [core_client.py:660] [shutdown] MPClient: cleaning up background resources (APIServer pid=1215265) INFO 07-24 12:12:45 [core_client.py:662] [shutdown] MPClient: complete (APIServer pid=1215265) INFO 07-24 12:12:45 [launcher.py:125] [shutdown] API server: engine client stopped (APIServer pid=1215265) INFO 07-24 12:12:45 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=1215265) INFO 07-24 12:12:45 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=1215265) INFO: Shutting down (APIServer pid=1215265) INFO: Waiting for application shutdown. (APIServer pid=1215265) INFO: Application shutdown complete. /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '