(APIServer pid=1142782) INFO 07-23 23:28:33 [api_utils.py:339] (APIServer pid=1142782) INFO 07-23 23:28:33 [api_utils.py:339] █ █ █▄ ▄█ (APIServer pid=1142782) INFO 07-23 23:28:33 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0 (APIServer pid=1142782) INFO 07-23 23:28:33 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/qwen35_pruned_keep25 (APIServer pid=1142782) INFO 07-23 23:28:33 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ (APIServer pid=1142782) INFO 07-23 23:28:33 [api_utils.py:339] (APIServer pid=1142782) INFO 07-23 23:28:33 [api_utils.py:273] non-default args: {'model_tag': 'outputs/qwen35_pruned_keep25', 'default_chat_template_kwargs': {'enable_thinking': False}, 'host': '127.0.0.1', 'port': 8399, 'model': 'outputs/qwen35_pruned_keep25', 'max_model_len': 8192, 'enforce_eager': True, 'served_model_name': ['student'], 'pipeline_parallel_size': 2, 'gpu_memory_utilization': 0.9} (APIServer pid=1142782) INFO 07-23 23:28:33 [model.py:619] Resolved architecture: PrunedQwen3_5MoeForCausalLM (APIServer pid=1142782) INFO 07-23 23:28:33 [model.py:1776] Using max model len 8192 (APIServer pid=1142782) INFO 07-23 23:28:33 [vllm.py:1042] Asynchronous scheduling is enabled. (APIServer pid=1142782) WARNING 07-23 23:28:33 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (APIServer pid=1142782) WARNING 07-23 23:28:33 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (APIServer pid=1142782) INFO 07-23 23:28:33 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) (APIServer pid=1142782) INFO 07-23 23:28:33 [vllm.py:1322] Cudagraph is disabled under eager mode (APIServer pid=1142782) INFO 07-23 23:28:33 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant (EngineCore pid=1142928) INFO 07-23 23:28:45 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/qwen35_pruned_keep25', speculative_config=None, tokenizer='outputs/qwen35_pruned_keep25', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=2, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto') (EngineCore pid=1142928) WARNING 07-23 23:28:45 [multiproc_executor.py:1067] Reducing Torch parallelism from 24 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. (EngineCore pid=1142928) INFO 07-23 23:28:45 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=192.168.0.15 (local), world_size=2, local_world_size=2 (Worker pid=1143042) INFO 07-23 23:28:53 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:57733 backend=nccl (Worker pid=1143043) INFO 07-23 23:28:53 [parallel_state.py:1607] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:57733 backend=nccl (Worker pid=1143042) INFO 07-23 23:28:54 [pynccl.py:113] vLLM is using nccl==2.28.9 (Worker pid=1143042) INFO 07-23 23:28:54 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'pp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. (Worker pid=1143042) INFO 07-23 23:28:54 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A (Worker pid=1143042) INFO 07-23 23:28:55 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. (Worker_PP0 pid=1143042) INFO 07-23 23:28:55 [gpu_model_runner.py:5209] Starting to load model outputs/qwen35_pruned_keep25... (Worker_PP1 pid=1143043) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. (Worker_PP1 pid=1143043) warnings.warn('Grouped GEMM not available.') (Worker_PP0 pid=1143042) INFO 07-23 23:28:55 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128). (Worker_PP0 pid=1143042) /home/henry/.cache/glean/megablocks-variable-93a1479bc15b/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. (Worker_PP0 pid=1143042) warnings.warn('Grouped GEMM not available.') (Worker_PP0 pid=1143042) INFO 07-23 23:28:55 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. (Worker_PP0 pid=1143042) INFO 07-23 23:28:55 [flash_attn.py:718] Using FlashAttention version 2 (Worker_PP0 pid=1143042) INFO 07-23 23:28:55 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 19.54 GiB. Available RAM: 107.13 GiB. (Worker_PP0 pid=1143042) INFO 07-23 23:28:55 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. (Worker_PP0 pid=1143042) Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00= mamba page size. (Worker_PP0 pid=1143042) INFO 07-23 23:29:02 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. (Worker_PP1 pid=1143043) INFO 07-23 23:29:03 [interface.py:890] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. (Worker_PP1 pid=1143043) INFO 07-23 23:29:03 [interface.py:914] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. (Worker_PP0 pid=1143042) INFO 07-23 23:29:04 [gpu_worker.py:538] Available KV cache memory: 11.07 GiB (EngineCore pid=1142928) INFO 07-23 23:29:05 [kv_cache_utils.py:2146] GPU KV cache size: 781,258 tokens (EngineCore pid=1142928) INFO 07-23 23:29:05 [kv_cache_utils.py:2147] Maximum concurrency for 8,192 tokens per request: 95.37x (Worker_PP1 pid=1143043) INFO 07-23 23:29:05 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. (Worker_PP0 pid=1143042) INFO 07-23 23:29:05 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested. (Worker_PP0 pid=1143042) INFO 07-23 23:29:05 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (Worker_PP1 pid=1143043) INFO 07-23 23:29:05 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn. (EngineCore pid=1142928) INFO 07-23 23:29:05 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.42 s (EngineCore pid=1142928) INFO 07-23 23:29:07 [vllm.py:1042] Asynchronous scheduling is enabled. (EngineCore pid=1142928) WARNING 07-23 23:29:07 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none (EngineCore pid=1142928) WARNING 07-23 23:29:07 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored. (EngineCore pid=1142928) INFO 07-23 23:29:07 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']) (EngineCore pid=1142928) INFO 07-23 23:29:07 [vllm.py:1322] Cudagraph is disabled under eager mode (EngineCore pid=1142928) INFO 07-23 23:29:07 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant (APIServer pid=1142782) INFO 07-23 23:29:07 [api_server.py:612] Supported tasks: ['generate'] (APIServer pid=1142782) INFO 07-23 23:29:09 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this. (APIServer pid=1142782) INFO 07-23 23:29:09 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8399 (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:37] Available routes are: (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /docs, Methods: HEAD, GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /redoc, Methods: HEAD, GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /load, Methods: GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /version, Methods: GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /health, Methods: GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /metrics, Methods: GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /tokenize, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /detokenize, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/models, Methods: GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /ping, Methods: GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /ping, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /invocations, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/chat/completions, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/responses, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/completions, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/messages, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /generative_scoring, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/completions/render, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /v1/completions/derender, Methods: POST (APIServer pid=1142782) INFO 07-23 23:29:09 [launcher.py:46] Route: /inference/v1/generate, Methods: POST (APIServer pid=1142782) INFO: Started server process [1142782] (APIServer pid=1142782) INFO: Waiting for application startup. (APIServer pid=1142782) INFO: Application startup complete. (APIServer pid=1142782) INFO: 127.0.0.1:44824 - "GET /health HTTP/1.1" 200 OK (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _zero_kv_blocks_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _fused_post_conv_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _build_route_rows. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) /home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/distributed/parallel_state.py:813: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_new.cpp:1586.) (Worker_PP0 pid=1143042) object_tensor = torch.frombuffer(pickle.dumps(obj), dtype=torch.uint8) [rank0]:[W723 23:29:42.382731493 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) [rank1]:[W723 23:29:42.382997731 ProcessGroupNCCL.cpp:4094] Warning: An unbatched P2P op (send/recv) was called on this ProcessGroup with size 2. In lazy initialization mode, this will result in a new 2-rank NCCL communicator to be created. (function operator()) (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _causal_conv1d_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_recurrent_gated_delta_rule_packed_decode_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: layer_norm_fwd_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (Worker_PP0 pid=1143042) WARNING 07-23 23:29:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: fused_sigmoid_gating_delta_rule_update_kernel. This causes a latency spike; consider extending warmup to cover this shape/config. (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:29:49 [loggers.py:273] Engine 000: Avg prompt throughput: 4906.3 tokens/s, Avg generation throughput: 29.8 tokens/s, Running: 17 reqs, Waiting: 46 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:29:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7106.2 tokens/s, Avg generation throughput: 83.3 tokens/s, Running: 26 reqs, Waiting: 36 reqs, GPU KV cache usage: 8.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:30:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6955.1 tokens/s, Avg generation throughput: 87.6 tokens/s, Running: 29 reqs, Waiting: 34 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:30:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7034.5 tokens/s, Avg generation throughput: 85.8 tokens/s, Running: 25 reqs, Waiting: 37 reqs, GPU KV cache usage: 8.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:30:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7087.0 tokens/s, Avg generation throughput: 89.9 tokens/s, Running: 23 reqs, Waiting: 37 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:30:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7126.8 tokens/s, Avg generation throughput: 81.3 tokens/s, Running: 27 reqs, Waiting: 35 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:30:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7026.2 tokens/s, Avg generation throughput: 81.6 tokens/s, Running: 25 reqs, Waiting: 35 reqs, GPU KV cache usage: 8.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:30:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6884.1 tokens/s, Avg generation throughput: 83.4 tokens/s, Running: 27 reqs, Waiting: 35 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:31:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7046.0 tokens/s, Avg generation throughput: 95.2 tokens/s, Running: 29 reqs, Waiting: 34 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:31:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7135.1 tokens/s, Avg generation throughput: 89.5 tokens/s, Running: 28 reqs, Waiting: 35 reqs, GPU KV cache usage: 10.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:31:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6905.5 tokens/s, Avg generation throughput: 102.4 tokens/s, Running: 29 reqs, Waiting: 33 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:31:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7062.1 tokens/s, Avg generation throughput: 94.2 tokens/s, Running: 30 reqs, Waiting: 31 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:31:49 [loggers.py:273] Engine 000: Avg prompt throughput: 6774.3 tokens/s, Avg generation throughput: 100.6 tokens/s, Running: 35 reqs, Waiting: 28 reqs, GPU KV cache usage: 12.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:31:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6873.6 tokens/s, Avg generation throughput: 124.3 tokens/s, Running: 37 reqs, Waiting: 25 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:32:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7023.6 tokens/s, Avg generation throughput: 106.4 tokens/s, Running: 28 reqs, Waiting: 35 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:32:19 [loggers.py:273] Engine 000: Avg prompt throughput: 6987.1 tokens/s, Avg generation throughput: 91.2 tokens/s, Running: 24 reqs, Waiting: 36 reqs, GPU KV cache usage: 8.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:32:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7072.7 tokens/s, Avg generation throughput: 82.9 tokens/s, Running: 36 reqs, Waiting: 27 reqs, GPU KV cache usage: 12.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:32:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7038.9 tokens/s, Avg generation throughput: 115.4 tokens/s, Running: 32 reqs, Waiting: 29 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:32:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7006.6 tokens/s, Avg generation throughput: 94.7 tokens/s, Running: 35 reqs, Waiting: 28 reqs, GPU KV cache usage: 11.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:32:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7113.6 tokens/s, Avg generation throughput: 124.3 tokens/s, Running: 33 reqs, Waiting: 24 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:33:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7038.9 tokens/s, Avg generation throughput: 114.5 tokens/s, Running: 36 reqs, Waiting: 24 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:33:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7043.1 tokens/s, Avg generation throughput: 114.8 tokens/s, Running: 36 reqs, Waiting: 26 reqs, GPU KV cache usage: 11.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:33:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7076.0 tokens/s, Avg generation throughput: 110.7 tokens/s, Running: 37 reqs, Waiting: 27 reqs, GPU KV cache usage: 12.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:33:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7012.9 tokens/s, Avg generation throughput: 111.3 tokens/s, Running: 33 reqs, Waiting: 27 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:33:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7059.1 tokens/s, Avg generation throughput: 108.1 tokens/s, Running: 35 reqs, Waiting: 26 reqs, GPU KV cache usage: 11.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:33:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7051.2 tokens/s, Avg generation throughput: 103.4 tokens/s, Running: 32 reqs, Waiting: 31 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57688 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:34:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6857.8 tokens/s, Avg generation throughput: 113.3 tokens/s, Running: 33 reqs, Waiting: 27 reqs, GPU KV cache usage: 10.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:34:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7308.2 tokens/s, Avg generation throughput: 99.6 tokens/s, Running: 25 reqs, Waiting: 36 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:34:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7050.4 tokens/s, Avg generation throughput: 83.2 tokens/s, Running: 18 reqs, Waiting: 43 reqs, GPU KV cache usage: 5.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:34:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7054.1 tokens/s, Avg generation throughput: 73.6 tokens/s, Running: 19 reqs, Waiting: 41 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:34:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7200.4 tokens/s, Avg generation throughput: 50.6 tokens/s, Running: 12 reqs, Waiting: 47 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:34:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7334.3 tokens/s, Avg generation throughput: 50.9 tokens/s, Running: 17 reqs, Waiting: 45 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:35:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6985.6 tokens/s, Avg generation throughput: 46.2 tokens/s, Running: 14 reqs, Waiting: 48 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:35:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7237.7 tokens/s, Avg generation throughput: 57.9 tokens/s, Running: 17 reqs, Waiting: 46 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:35:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7010.9 tokens/s, Avg generation throughput: 50.4 tokens/s, Running: 15 reqs, Waiting: 48 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:35:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7125.2 tokens/s, Avg generation throughput: 49.6 tokens/s, Running: 14 reqs, Waiting: 48 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:35:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7428.8 tokens/s, Avg generation throughput: 48.7 tokens/s, Running: 17 reqs, Waiting: 43 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:35:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7056.1 tokens/s, Avg generation throughput: 57.1 tokens/s, Running: 15 reqs, Waiting: 45 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:36:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7122.5 tokens/s, Avg generation throughput: 45.6 tokens/s, Running: 15 reqs, Waiting: 48 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:36:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7109.2 tokens/s, Avg generation throughput: 47.1 tokens/s, Running: 16 reqs, Waiting: 48 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:36:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7219.0 tokens/s, Avg generation throughput: 52.4 tokens/s, Running: 17 reqs, Waiting: 46 reqs, GPU KV cache usage: 6.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:36:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7216.3 tokens/s, Avg generation throughput: 53.4 tokens/s, Running: 14 reqs, Waiting: 47 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:36:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7073.5 tokens/s, Avg generation throughput: 40.3 tokens/s, Running: 15 reqs, Waiting: 46 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:36:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7126.9 tokens/s, Avg generation throughput: 44.5 tokens/s, Running: 13 reqs, Waiting: 47 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:37:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7355.6 tokens/s, Avg generation throughput: 51.0 tokens/s, Running: 17 reqs, Waiting: 45 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:37:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7152.7 tokens/s, Avg generation throughput: 50.5 tokens/s, Running: 17 reqs, Waiting: 43 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:37:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7002.2 tokens/s, Avg generation throughput: 52.4 tokens/s, Running: 14 reqs, Waiting: 48 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:37:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7117.2 tokens/s, Avg generation throughput: 54.5 tokens/s, Running: 17 reqs, Waiting: 45 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:37:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7201.8 tokens/s, Avg generation throughput: 56.6 tokens/s, Running: 16 reqs, Waiting: 43 reqs, GPU KV cache usage: 6.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:37:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7085.7 tokens/s, Avg generation throughput: 53.8 tokens/s, Running: 13 reqs, Waiting: 49 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:38:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7137.1 tokens/s, Avg generation throughput: 53.4 tokens/s, Running: 18 reqs, Waiting: 44 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:38:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7164.0 tokens/s, Avg generation throughput: 53.1 tokens/s, Running: 18 reqs, Waiting: 44 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:38:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6978.8 tokens/s, Avg generation throughput: 61.3 tokens/s, Running: 18 reqs, Waiting: 45 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:38:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7317.7 tokens/s, Avg generation throughput: 50.4 tokens/s, Running: 15 reqs, Waiting: 47 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:38:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7147.9 tokens/s, Avg generation throughput: 54.5 tokens/s, Running: 19 reqs, Waiting: 43 reqs, GPU KV cache usage: 7.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:38:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7140.2 tokens/s, Avg generation throughput: 52.5 tokens/s, Running: 18 reqs, Waiting: 45 reqs, GPU KV cache usage: 6.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:39:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7150.8 tokens/s, Avg generation throughput: 50.9 tokens/s, Running: 13 reqs, Waiting: 51 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:39:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7138.5 tokens/s, Avg generation throughput: 42.2 tokens/s, Running: 12 reqs, Waiting: 50 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:39:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7339.2 tokens/s, Avg generation throughput: 40.1 tokens/s, Running: 10 reqs, Waiting: 51 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:39:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7068.2 tokens/s, Avg generation throughput: 40.7 tokens/s, Running: 13 reqs, Waiting: 47 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:39:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7392.6 tokens/s, Avg generation throughput: 47.4 tokens/s, Running: 12 reqs, Waiting: 49 reqs, GPU KV cache usage: 4.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:39:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7063.0 tokens/s, Avg generation throughput: 43.9 tokens/s, Running: 12 reqs, Waiting: 50 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:40:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7333.9 tokens/s, Avg generation throughput: 41.3 tokens/s, Running: 15 reqs, Waiting: 48 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:40:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7114.3 tokens/s, Avg generation throughput: 40.0 tokens/s, Running: 11 reqs, Waiting: 48 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:40:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7384.8 tokens/s, Avg generation throughput: 39.2 tokens/s, Running: 14 reqs, Waiting: 49 reqs, GPU KV cache usage: 5.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:40:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7044.9 tokens/s, Avg generation throughput: 38.0 tokens/s, Running: 11 reqs, Waiting: 51 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:40:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7351.9 tokens/s, Avg generation throughput: 42.7 tokens/s, Running: 16 reqs, Waiting: 43 reqs, GPU KV cache usage: 5.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:40:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7127.2 tokens/s, Avg generation throughput: 46.6 tokens/s, Running: 18 reqs, Waiting: 43 reqs, GPU KV cache usage: 5.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:41:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7409.5 tokens/s, Avg generation throughput: 53.8 tokens/s, Running: 14 reqs, Waiting: 47 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:41:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7108.8 tokens/s, Avg generation throughput: 51.0 tokens/s, Running: 11 reqs, Waiting: 50 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:41:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7274.6 tokens/s, Avg generation throughput: 60.0 tokens/s, Running: 16 reqs, Waiting: 45 reqs, GPU KV cache usage: 5.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:41:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7125.0 tokens/s, Avg generation throughput: 50.6 tokens/s, Running: 17 reqs, Waiting: 44 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:41:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7346.4 tokens/s, Avg generation throughput: 52.5 tokens/s, Running: 15 reqs, Waiting: 47 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:41:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7062.9 tokens/s, Avg generation throughput: 47.0 tokens/s, Running: 17 reqs, Waiting: 43 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:42:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7355.0 tokens/s, Avg generation throughput: 59.6 tokens/s, Running: 19 reqs, Waiting: 44 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:42:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7069.5 tokens/s, Avg generation throughput: 61.0 tokens/s, Running: 17 reqs, Waiting: 45 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:42:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7079.9 tokens/s, Avg generation throughput: 56.8 tokens/s, Running: 19 reqs, Waiting: 44 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:42:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7274.3 tokens/s, Avg generation throughput: 73.8 tokens/s, Running: 21 reqs, Waiting: 36 reqs, GPU KV cache usage: 6.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:42:49 [loggers.py:273] Engine 000: Avg prompt throughput: 7215.4 tokens/s, Avg generation throughput: 60.5 tokens/s, Running: 16 reqs, Waiting: 45 reqs, GPU KV cache usage: 5.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:42:59 [loggers.py:273] Engine 000: Avg prompt throughput: 7110.1 tokens/s, Avg generation throughput: 54.9 tokens/s, Running: 18 reqs, Waiting: 44 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:43:09 [loggers.py:273] Engine 000: Avg prompt throughput: 7305.5 tokens/s, Avg generation throughput: 57.1 tokens/s, Running: 21 reqs, Waiting: 42 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:43:19 [loggers.py:273] Engine 000: Avg prompt throughput: 7000.5 tokens/s, Avg generation throughput: 64.9 tokens/s, Running: 15 reqs, Waiting: 46 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:43:29 [loggers.py:273] Engine 000: Avg prompt throughput: 7179.1 tokens/s, Avg generation throughput: 69.3 tokens/s, Running: 27 reqs, Waiting: 33 reqs, GPU KV cache usage: 9.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:43:39 [loggers.py:273] Engine 000: Avg prompt throughput: 6986.1 tokens/s, Avg generation throughput: 124.7 tokens/s, Running: 43 reqs, Waiting: 21 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57554 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:43:49 [loggers.py:273] Engine 000: Avg prompt throughput: 6855.3 tokens/s, Avg generation throughput: 153.0 tokens/s, Running: 49 reqs, Waiting: 13 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:43:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6541.8 tokens/s, Avg generation throughput: 184.4 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:44:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6214.1 tokens/s, Avg generation throughput: 205.7 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:44:19 [loggers.py:273] Engine 000: Avg prompt throughput: 6630.7 tokens/s, Avg generation throughput: 177.1 tokens/s, Running: 53 reqs, Waiting: 8 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:44:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6765.2 tokens/s, Avg generation throughput: 183.8 tokens/s, Running: 57 reqs, Waiting: 3 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:44:39 [loggers.py:273] Engine 000: Avg prompt throughput: 6651.6 tokens/s, Avg generation throughput: 178.0 tokens/s, Running: 51 reqs, Waiting: 8 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:44:49 [loggers.py:273] Engine 000: Avg prompt throughput: 6588.5 tokens/s, Avg generation throughput: 187.6 tokens/s, Running: 53 reqs, Waiting: 10 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:44:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6739.9 tokens/s, Avg generation throughput: 166.8 tokens/s, Running: 54 reqs, Waiting: 8 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:45:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6778.7 tokens/s, Avg generation throughput: 173.1 tokens/s, Running: 49 reqs, Waiting: 13 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:45:19 [loggers.py:273] Engine 000: Avg prompt throughput: 6748.0 tokens/s, Avg generation throughput: 173.9 tokens/s, Running: 55 reqs, Waiting: 8 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:45:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6622.3 tokens/s, Avg generation throughput: 179.1 tokens/s, Running: 56 reqs, Waiting: 8 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:45:39 [loggers.py:273] Engine 000: Avg prompt throughput: 6500.7 tokens/s, Avg generation throughput: 190.7 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57580 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:45:49 [loggers.py:273] Engine 000: Avg prompt throughput: 6613.8 tokens/s, Avg generation throughput: 183.1 tokens/s, Running: 57 reqs, Waiting: 6 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:45:59 [loggers.py:273] Engine 000: Avg prompt throughput: 6664.4 tokens/s, Avg generation throughput: 185.0 tokens/s, Running: 52 reqs, Waiting: 8 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:46:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6765.8 tokens/s, Avg generation throughput: 186.4 tokens/s, Running: 58 reqs, Waiting: 2 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:46:19 [loggers.py:273] Engine 000: Avg prompt throughput: 6602.4 tokens/s, Avg generation throughput: 173.6 tokens/s, Running: 51 reqs, Waiting: 11 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:46:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6797.4 tokens/s, Avg generation throughput: 172.1 tokens/s, Running: 52 reqs, Waiting: 11 reqs, GPU KV cache usage: 17.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:46:39 [loggers.py:273] Engine 000: Avg prompt throughput: 6576.1 tokens/s, Avg generation throughput: 168.5 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:46:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6812.3 tokens/s, Avg generation throughput: 174.2 tokens/s, Running: 48 reqs, Waiting: 12 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:47:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6517.2 tokens/s, Avg generation throughput: 180.4 tokens/s, Running: 60 reqs, Waiting: 1 reqs, GPU KV cache usage: 20.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:47:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6752.5 tokens/s, Avg generation throughput: 185.2 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:47:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6893.4 tokens/s, Avg generation throughput: 178.5 tokens/s, Running: 58 reqs, Waiting: 3 reqs, GPU KV cache usage: 19.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:47:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6649.2 tokens/s, Avg generation throughput: 186.1 tokens/s, Running: 60 reqs, Waiting: 1 reqs, GPU KV cache usage: 20.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:47:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6896.5 tokens/s, Avg generation throughput: 167.7 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:47:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6750.4 tokens/s, Avg generation throughput: 135.7 tokens/s, Running: 39 reqs, Waiting: 21 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:48:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7059.0 tokens/s, Avg generation throughput: 148.5 tokens/s, Running: 48 reqs, Waiting: 16 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:48:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6829.5 tokens/s, Avg generation throughput: 147.2 tokens/s, Running: 45 reqs, Waiting: 16 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:48:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6849.4 tokens/s, Avg generation throughput: 154.2 tokens/s, Running: 48 reqs, Waiting: 12 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:48:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6706.1 tokens/s, Avg generation throughput: 168.6 tokens/s, Running: 51 reqs, Waiting: 10 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:48:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6768.3 tokens/s, Avg generation throughput: 169.5 tokens/s, Running: 50 reqs, Waiting: 13 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:48:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6923.4 tokens/s, Avg generation throughput: 156.5 tokens/s, Running: 46 reqs, Waiting: 16 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:49:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6747.1 tokens/s, Avg generation throughput: 146.5 tokens/s, Running: 44 reqs, Waiting: 17 reqs, GPU KV cache usage: 14.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:49:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7051.9 tokens/s, Avg generation throughput: 149.7 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:49:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6802.4 tokens/s, Avg generation throughput: 143.6 tokens/s, Running: 40 reqs, Waiting: 20 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57660 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:49:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6814.4 tokens/s, Avg generation throughput: 130.2 tokens/s, Running: 33 reqs, Waiting: 31 reqs, GPU KV cache usage: 12.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:49:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6765.2 tokens/s, Avg generation throughput: 130.8 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:49:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6478.1 tokens/s, Avg generation throughput: 142.5 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:50:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6874.8 tokens/s, Avg generation throughput: 141.4 tokens/s, Running: 42 reqs, Waiting: 20 reqs, GPU KV cache usage: 18.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57592 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:50:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6513.2 tokens/s, Avg generation throughput: 138.2 tokens/s, Running: 43 reqs, Waiting: 16 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57416 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:50:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6456.1 tokens/s, Avg generation throughput: 136.5 tokens/s, Running: 45 reqs, Waiting: 16 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57948 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:50:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6460.5 tokens/s, Avg generation throughput: 145.3 tokens/s, Running: 43 reqs, Waiting: 19 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:50:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6820.5 tokens/s, Avg generation throughput: 144.8 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 19.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:50:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6536.9 tokens/s, Avg generation throughput: 141.3 tokens/s, Running: 44 reqs, Waiting: 18 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:51:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6706.1 tokens/s, Avg generation throughput: 140.0 tokens/s, Running: 41 reqs, Waiting: 22 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57392 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:51:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6638.5 tokens/s, Avg generation throughput: 132.6 tokens/s, Running: 41 reqs, Waiting: 22 reqs, GPU KV cache usage: 17.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:51:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6605.0 tokens/s, Avg generation throughput: 139.9 tokens/s, Running: 44 reqs, Waiting: 19 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:51:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6580.3 tokens/s, Avg generation throughput: 137.6 tokens/s, Running: 40 reqs, Waiting: 23 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:51:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6628.9 tokens/s, Avg generation throughput: 131.8 tokens/s, Running: 46 reqs, Waiting: 18 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:51:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6311.3 tokens/s, Avg generation throughput: 166.6 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 21.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:52:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6571.8 tokens/s, Avg generation throughput: 182.5 tokens/s, Running: 57 reqs, Waiting: 7 reqs, GPU KV cache usage: 21.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:52:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6536.9 tokens/s, Avg generation throughput: 199.9 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:52:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6455.1 tokens/s, Avg generation throughput: 192.7 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 22.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:52:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6035.2 tokens/s, Avg generation throughput: 210.6 tokens/s, Running: 60 reqs, Waiting: 2 reqs, GPU KV cache usage: 22.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:52:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6382.6 tokens/s, Avg generation throughput: 191.5 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 22.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:52:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6259.3 tokens/s, Avg generation throughput: 210.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:53:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6243.6 tokens/s, Avg generation throughput: 204.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:53:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6246.9 tokens/s, Avg generation throughput: 204.8 tokens/s, Running: 63 reqs, Waiting: 1 reqs, GPU KV cache usage: 23.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:53:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6477.8 tokens/s, Avg generation throughput: 210.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 23.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:53:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6022.5 tokens/s, Avg generation throughput: 206.1 tokens/s, Running: 54 reqs, Waiting: 6 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:53:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6591.8 tokens/s, Avg generation throughput: 175.1 tokens/s, Running: 56 reqs, Waiting: 7 reqs, GPU KV cache usage: 21.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:53:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6632.5 tokens/s, Avg generation throughput: 169.6 tokens/s, Running: 56 reqs, Waiting: 4 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:54:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6613.2 tokens/s, Avg generation throughput: 169.6 tokens/s, Running: 52 reqs, Waiting: 10 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:54:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6471.0 tokens/s, Avg generation throughput: 173.7 tokens/s, Running: 53 reqs, Waiting: 8 reqs, GPU KV cache usage: 20.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:54:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6674.9 tokens/s, Avg generation throughput: 181.4 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 20.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:54:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6792.0 tokens/s, Avg generation throughput: 168.4 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:54:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6474.4 tokens/s, Avg generation throughput: 179.3 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 22.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:54:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6644.8 tokens/s, Avg generation throughput: 186.0 tokens/s, Running: 58 reqs, Waiting: 5 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:55:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6341.2 tokens/s, Avg generation throughput: 190.6 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:55:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6604.2 tokens/s, Avg generation throughput: 171.9 tokens/s, Running: 55 reqs, Waiting: 7 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:55:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6819.5 tokens/s, Avg generation throughput: 187.1 tokens/s, Running: 54 reqs, Waiting: 7 reqs, GPU KV cache usage: 19.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:55:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6508.0 tokens/s, Avg generation throughput: 184.7 tokens/s, Running: 55 reqs, Waiting: 5 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:55:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6607.1 tokens/s, Avg generation throughput: 172.9 tokens/s, Running: 59 reqs, Waiting: 5 reqs, GPU KV cache usage: 21.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:55:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6581.5 tokens/s, Avg generation throughput: 179.8 tokens/s, Running: 50 reqs, Waiting: 12 reqs, GPU KV cache usage: 18.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:56:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6806.9 tokens/s, Avg generation throughput: 155.6 tokens/s, Running: 48 reqs, Waiting: 13 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:56:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6624.9 tokens/s, Avg generation throughput: 166.8 tokens/s, Running: 48 reqs, Waiting: 14 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:56:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6691.9 tokens/s, Avg generation throughput: 113.2 tokens/s, Running: 21 reqs, Waiting: 41 reqs, GPU KV cache usage: 8.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:56:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7170.5 tokens/s, Avg generation throughput: 45.3 tokens/s, Running: 12 reqs, Waiting: 49 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:56:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7315.7 tokens/s, Avg generation throughput: 44.5 tokens/s, Running: 15 reqs, Waiting: 46 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:56:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7141.5 tokens/s, Avg generation throughput: 42.2 tokens/s, Running: 10 reqs, Waiting: 49 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:57:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7205.3 tokens/s, Avg generation throughput: 32.8 tokens/s, Running: 12 reqs, Waiting: 50 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:57:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6997.6 tokens/s, Avg generation throughput: 41.5 tokens/s, Running: 12 reqs, Waiting: 52 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:57:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7349.9 tokens/s, Avg generation throughput: 34.6 tokens/s, Running: 9 reqs, Waiting: 53 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:57:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7295.6 tokens/s, Avg generation throughput: 33.7 tokens/s, Running: 14 reqs, Waiting: 49 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:57:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7049.1 tokens/s, Avg generation throughput: 33.7 tokens/s, Running: 7 reqs, Waiting: 55 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:57:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7325.2 tokens/s, Avg generation throughput: 34.2 tokens/s, Running: 10 reqs, Waiting: 51 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:58:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7230.2 tokens/s, Avg generation throughput: 42.1 tokens/s, Running: 12 reqs, Waiting: 48 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:58:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7133.7 tokens/s, Avg generation throughput: 28.5 tokens/s, Running: 8 reqs, Waiting: 55 reqs, GPU KV cache usage: 2.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:58:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7303.8 tokens/s, Avg generation throughput: 33.0 tokens/s, Running: 12 reqs, Waiting: 50 reqs, GPU KV cache usage: 4.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:58:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7033.5 tokens/s, Avg generation throughput: 31.7 tokens/s, Running: 10 reqs, Waiting: 52 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:58:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7251.3 tokens/s, Avg generation throughput: 37.5 tokens/s, Running: 14 reqs, Waiting: 49 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:58:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6980.2 tokens/s, Avg generation throughput: 41.1 tokens/s, Running: 13 reqs, Waiting: 49 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:59:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7447.1 tokens/s, Avg generation throughput: 35.5 tokens/s, Running: 9 reqs, Waiting: 53 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:59:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7063.5 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 12 reqs, Waiting: 50 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:59:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7162.5 tokens/s, Avg generation throughput: 34.3 tokens/s, Running: 9 reqs, Waiting: 53 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:59:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7365.7 tokens/s, Avg generation throughput: 39.0 tokens/s, Running: 15 reqs, Waiting: 47 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:59:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7158.4 tokens/s, Avg generation throughput: 39.0 tokens/s, Running: 11 reqs, Waiting: 51 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-23 23:59:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7160.6 tokens/s, Avg generation throughput: 40.6 tokens/s, Running: 11 reqs, Waiting: 51 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:00:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7123.8 tokens/s, Avg generation throughput: 28.7 tokens/s, Running: 10 reqs, Waiting: 53 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:00:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7353.0 tokens/s, Avg generation throughput: 34.5 tokens/s, Running: 13 reqs, Waiting: 50 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:00:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7115.9 tokens/s, Avg generation throughput: 36.8 tokens/s, Running: 14 reqs, Waiting: 49 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:00:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6974.9 tokens/s, Avg generation throughput: 39.3 tokens/s, Running: 10 reqs, Waiting: 52 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:00:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7506.8 tokens/s, Avg generation throughput: 32.7 tokens/s, Running: 9 reqs, Waiting: 53 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:00:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7032.9 tokens/s, Avg generation throughput: 28.4 tokens/s, Running: 10 reqs, Waiting: 53 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:01:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7127.5 tokens/s, Avg generation throughput: 34.9 tokens/s, Running: 12 reqs, Waiting: 49 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:01:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7328.0 tokens/s, Avg generation throughput: 38.3 tokens/s, Running: 13 reqs, Waiting: 49 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:01:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7096.1 tokens/s, Avg generation throughput: 35.2 tokens/s, Running: 9 reqs, Waiting: 54 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:01:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7189.5 tokens/s, Avg generation throughput: 37.2 tokens/s, Running: 9 reqs, Waiting: 52 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:01:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7215.7 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 13 reqs, Waiting: 48 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:01:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7266.4 tokens/s, Avg generation throughput: 40.6 tokens/s, Running: 13 reqs, Waiting: 50 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:02:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7140.4 tokens/s, Avg generation throughput: 30.6 tokens/s, Running: 9 reqs, Waiting: 53 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57820 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:02:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7299.7 tokens/s, Avg generation throughput: 43.0 tokens/s, Running: 29 reqs, Waiting: 35 reqs, GPU KV cache usage: 8.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:02:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6142.1 tokens/s, Avg generation throughput: 221.4 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:02:30 [loggers.py:273] Engine 000: Avg prompt throughput: 5811.8 tokens/s, Avg generation throughput: 254.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57448 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:02:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6051.6 tokens/s, Avg generation throughput: 254.2 tokens/s, Running: 61 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:02:50 [loggers.py:273] Engine 000: Avg prompt throughput: 5681.1 tokens/s, Avg generation throughput: 240.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:03:00 [loggers.py:273] Engine 000: Avg prompt throughput: 5603.7 tokens/s, Avg generation throughput: 260.0 tokens/s, Running: 58 reqs, Waiting: 2 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59284 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:03:10 [loggers.py:273] Engine 000: Avg prompt throughput: 5796.6 tokens/s, Avg generation throughput: 267.5 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:03:20 [loggers.py:273] Engine 000: Avg prompt throughput: 5587.7 tokens/s, Avg generation throughput: 241.3 tokens/s, Running: 60 reqs, Waiting: 1 reqs, GPU KV cache usage: 17.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:03:30 [loggers.py:273] Engine 000: Avg prompt throughput: 5913.6 tokens/s, Avg generation throughput: 259.8 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:03:40 [loggers.py:273] Engine 000: Avg prompt throughput: 5553.4 tokens/s, Avg generation throughput: 259.9 tokens/s, Running: 61 reqs, Waiting: 2 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:03:50 [loggers.py:273] Engine 000: Avg prompt throughput: 5577.2 tokens/s, Avg generation throughput: 268.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57374 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:04:00 [loggers.py:273] Engine 000: Avg prompt throughput: 5637.7 tokens/s, Avg generation throughput: 248.3 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57430 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:04:10 [loggers.py:273] Engine 000: Avg prompt throughput: 5750.3 tokens/s, Avg generation throughput: 268.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:04:20 [loggers.py:273] Engine 000: Avg prompt throughput: 5518.2 tokens/s, Avg generation throughput: 261.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:04:30 [loggers.py:273] Engine 000: Avg prompt throughput: 5820.4 tokens/s, Avg generation throughput: 267.4 tokens/s, Running: 61 reqs, Waiting: 1 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:04:40 [loggers.py:273] Engine 000: Avg prompt throughput: 5361.1 tokens/s, Avg generation throughput: 268.8 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:04:50 [loggers.py:273] Engine 000: Avg prompt throughput: 5670.5 tokens/s, Avg generation throughput: 274.3 tokens/s, Running: 60 reqs, Waiting: 1 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:05:00 [loggers.py:273] Engine 000: Avg prompt throughput: 5423.3 tokens/s, Avg generation throughput: 263.1 tokens/s, Running: 50 reqs, Waiting: 9 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:05:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7006.3 tokens/s, Avg generation throughput: 143.2 tokens/s, Running: 37 reqs, Waiting: 24 reqs, GPU KV cache usage: 12.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:05:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6807.5 tokens/s, Avg generation throughput: 132.8 tokens/s, Running: 39 reqs, Waiting: 20 reqs, GPU KV cache usage: 12.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:05:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6999.4 tokens/s, Avg generation throughput: 136.4 tokens/s, Running: 42 reqs, Waiting: 19 reqs, GPU KV cache usage: 13.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:05:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6858.4 tokens/s, Avg generation throughput: 152.6 tokens/s, Running: 51 reqs, Waiting: 13 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:05:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6831.7 tokens/s, Avg generation throughput: 174.1 tokens/s, Running: 53 reqs, Waiting: 6 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:06:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6714.0 tokens/s, Avg generation throughput: 180.0 tokens/s, Running: 52 reqs, Waiting: 9 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:06:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6824.1 tokens/s, Avg generation throughput: 175.7 tokens/s, Running: 52 reqs, Waiting: 8 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:06:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6807.1 tokens/s, Avg generation throughput: 182.8 tokens/s, Running: 48 reqs, Waiting: 10 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:06:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6688.3 tokens/s, Avg generation throughput: 152.9 tokens/s, Running: 45 reqs, Waiting: 17 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:06:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6948.3 tokens/s, Avg generation throughput: 139.0 tokens/s, Running: 37 reqs, Waiting: 23 reqs, GPU KV cache usage: 12.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:06:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6973.7 tokens/s, Avg generation throughput: 117.3 tokens/s, Running: 34 reqs, Waiting: 26 reqs, GPU KV cache usage: 11.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:07:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6849.1 tokens/s, Avg generation throughput: 105.0 tokens/s, Running: 32 reqs, Waiting: 30 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:07:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7043.9 tokens/s, Avg generation throughput: 105.8 tokens/s, Running: 27 reqs, Waiting: 31 reqs, GPU KV cache usage: 9.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:07:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7062.6 tokens/s, Avg generation throughput: 99.5 tokens/s, Running: 28 reqs, Waiting: 33 reqs, GPU KV cache usage: 9.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:07:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7178.1 tokens/s, Avg generation throughput: 95.4 tokens/s, Running: 24 reqs, Waiting: 37 reqs, GPU KV cache usage: 8.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:07:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6877.5 tokens/s, Avg generation throughput: 85.8 tokens/s, Running: 26 reqs, Waiting: 37 reqs, GPU KV cache usage: 8.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:07:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6951.5 tokens/s, Avg generation throughput: 84.9 tokens/s, Running: 21 reqs, Waiting: 42 reqs, GPU KV cache usage: 7.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:08:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7145.6 tokens/s, Avg generation throughput: 76.8 tokens/s, Running: 22 reqs, Waiting: 38 reqs, GPU KV cache usage: 7.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:08:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7169.3 tokens/s, Avg generation throughput: 75.4 tokens/s, Running: 19 reqs, Waiting: 42 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:08:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6982.3 tokens/s, Avg generation throughput: 70.5 tokens/s, Running: 21 reqs, Waiting: 40 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:08:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7180.4 tokens/s, Avg generation throughput: 64.7 tokens/s, Running: 19 reqs, Waiting: 41 reqs, GPU KV cache usage: 6.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:08:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7085.3 tokens/s, Avg generation throughput: 67.3 tokens/s, Running: 22 reqs, Waiting: 39 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:08:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7346.7 tokens/s, Avg generation throughput: 60.7 tokens/s, Running: 21 reqs, Waiting: 41 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:09:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7084.4 tokens/s, Avg generation throughput: 54.3 tokens/s, Running: 18 reqs, Waiting: 42 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:09:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7065.6 tokens/s, Avg generation throughput: 60.3 tokens/s, Running: 15 reqs, Waiting: 45 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:09:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7159.1 tokens/s, Avg generation throughput: 51.5 tokens/s, Running: 19 reqs, Waiting: 42 reqs, GPU KV cache usage: 6.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:09:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7320.0 tokens/s, Avg generation throughput: 68.2 tokens/s, Running: 18 reqs, Waiting: 44 reqs, GPU KV cache usage: 5.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:09:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7061.5 tokens/s, Avg generation throughput: 57.8 tokens/s, Running: 17 reqs, Waiting: 45 reqs, GPU KV cache usage: 5.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:09:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7102.1 tokens/s, Avg generation throughput: 60.8 tokens/s, Running: 20 reqs, Waiting: 40 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:10:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7048.0 tokens/s, Avg generation throughput: 69.0 tokens/s, Running: 22 reqs, Waiting: 40 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:10:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7339.9 tokens/s, Avg generation throughput: 69.3 tokens/s, Running: 17 reqs, Waiting: 46 reqs, GPU KV cache usage: 5.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:10:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7117.1 tokens/s, Avg generation throughput: 56.2 tokens/s, Running: 16 reqs, Waiting: 46 reqs, GPU KV cache usage: 5.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:10:30 [loggers.py:273] Engine 000: Avg prompt throughput: 7083.8 tokens/s, Avg generation throughput: 73.0 tokens/s, Running: 33 reqs, Waiting: 30 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:10:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7073.1 tokens/s, Avg generation throughput: 164.8 tokens/s, Running: 56 reqs, Waiting: 2 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:10:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6764.0 tokens/s, Avg generation throughput: 186.1 tokens/s, Running: 57 reqs, Waiting: 5 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:11:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6797.4 tokens/s, Avg generation throughput: 186.7 tokens/s, Running: 52 reqs, Waiting: 7 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:11:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6731.4 tokens/s, Avg generation throughput: 175.0 tokens/s, Running: 55 reqs, Waiting: 5 reqs, GPU KV cache usage: 18.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:11:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6474.2 tokens/s, Avg generation throughput: 192.2 tokens/s, Running: 55 reqs, Waiting: 4 reqs, GPU KV cache usage: 18.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:11:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6762.6 tokens/s, Avg generation throughput: 182.2 tokens/s, Running: 60 reqs, Waiting: 3 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:11:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6762.7 tokens/s, Avg generation throughput: 191.3 tokens/s, Running: 62 reqs, Waiting: 1 reqs, GPU KV cache usage: 20.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:11:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6625.0 tokens/s, Avg generation throughput: 203.1 tokens/s, Running: 58 reqs, Waiting: 4 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:12:00 [loggers.py:273] Engine 000: Avg prompt throughput: 6880.0 tokens/s, Avg generation throughput: 189.6 tokens/s, Running: 59 reqs, Waiting: 1 reqs, GPU KV cache usage: 19.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:12:10 [loggers.py:273] Engine 000: Avg prompt throughput: 6562.4 tokens/s, Avg generation throughput: 192.4 tokens/s, Running: 56 reqs, Waiting: 4 reqs, GPU KV cache usage: 18.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:12:20 [loggers.py:273] Engine 000: Avg prompt throughput: 6338.0 tokens/s, Avg generation throughput: 199.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:12:30 [loggers.py:273] Engine 000: Avg prompt throughput: 6694.5 tokens/s, Avg generation throughput: 190.2 tokens/s, Running: 56 reqs, Waiting: 6 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:12:40 [loggers.py:273] Engine 000: Avg prompt throughput: 6350.2 tokens/s, Avg generation throughput: 207.0 tokens/s, Running: 56 reqs, Waiting: 5 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57624 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57676 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57566 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:42588 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:39182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57462 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57386 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:40388 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57584 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57596 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57788 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57506 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:56498 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:43656 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57710 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57850 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57452 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:12:50 [loggers.py:273] Engine 000: Avg prompt throughput: 6111.9 tokens/s, Avg generation throughput: 219.9 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57842 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57378 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57816 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57640 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57762 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45356 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57474 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57698 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57522 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57604 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57472 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57706 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57444 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57752 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:41314 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:54046 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57928 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:13:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 194.8 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57742 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:13:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 141.3 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:59070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:13:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 101.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:13:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 99.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:13:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 99.0 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:13:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 99.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:14:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 99.9 tokens/s, Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:56092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:14:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 91.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57408 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:14:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 79.6 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:14:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 77.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:14:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 77.7 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:57528 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57490 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57458 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:57368 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34186 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:14:50 [loggers.py:273] Engine 000: Avg prompt throughput: 1855.6 tokens/s, Avg generation throughput: 214.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:34186 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33690 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34186 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:15:00 [loggers.py:273] Engine 000: Avg prompt throughput: 723.0 tokens/s, Avg generation throughput: 615.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34230 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:15:10 [loggers.py:273] Engine 000: Avg prompt throughput: 758.2 tokens/s, Avg generation throughput: 609.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:15:20 [loggers.py:273] Engine 000: Avg prompt throughput: 735.1 tokens/s, Avg generation throughput: 607.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:34132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:15:30 [loggers.py:273] Engine 000: Avg prompt throughput: 520.7 tokens/s, Avg generation throughput: 629.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:15:40 [loggers.py:273] Engine 000: Avg prompt throughput: 167.9 tokens/s, Avg generation throughput: 670.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34204 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33914 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34038 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34156 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:15:50 [loggers.py:273] Engine 000: Avg prompt throughput: 104.2 tokens/s, Avg generation throughput: 651.5 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33732 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:16:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 583.1 tokens/s, Running: 53 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:34052 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:16:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 567.3 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:16:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 566.7 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:16:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 561.5 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:34072 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:16:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 554.5 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33694 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33708 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33714 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33814 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33872 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33888 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33898 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:16:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 536.9 tokens/s, Running: 40 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34064 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34088 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34176 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34208 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33924 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34172 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33690 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34186 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33974 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33828 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:17:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 275.6 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33838 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34040 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34230 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34198 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:17:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 216.1 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34092 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33982 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33730 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34216 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:17:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 140.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:34020 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34122 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34144 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:34028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:17:30 [loggers.py:273] Engine 000: Avg prompt throughput: 75.6 tokens/s, Avg generation throughput: 40.4 tokens/s, Running: 3 reqs, Waiting: 60 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44636 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44824 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:17:40 [loggers.py:273] Engine 000: Avg prompt throughput: 7328.2 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 17 reqs, Waiting: 43 reqs, GPU KV cache usage: 4.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44824 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44824 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:17:50 [loggers.py:273] Engine 000: Avg prompt throughput: 7333.9 tokens/s, Avg generation throughput: 43.1 tokens/s, Running: 13 reqs, Waiting: 49 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44636 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44824 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44636 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44824 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:18:00 [loggers.py:273] Engine 000: Avg prompt throughput: 7305.7 tokens/s, Avg generation throughput: 34.1 tokens/s, Running: 10 reqs, Waiting: 50 reqs, GPU KV cache usage: 2.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44636 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:18:10 [loggers.py:273] Engine 000: Avg prompt throughput: 7291.7 tokens/s, Avg generation throughput: 44.7 tokens/s, Running: 14 reqs, Waiting: 42 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44636 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44636 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44792 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44844 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44880 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44870 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44800 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44766 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44740 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44810 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44918 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44836 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44720 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44824 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44922 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45012 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44970 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:18:20 [loggers.py:273] Engine 000: Avg prompt throughput: 7377.1 tokens/s, Avg generation throughput: 46.2 tokens/s, Running: 17 reqs, Waiting: 27 reqs, GPU KV cache usage: 4.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44992 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44808 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44650 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44602 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44758 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44946 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44518 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44468 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44496 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44934 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44704 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44664 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44544 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44560 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45000 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44538 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44682 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44546 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44576 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44652 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44598 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44674 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44618 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44590 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44636 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44686 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44520 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44512 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44534 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44716 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44754 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44734 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44718 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44700 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44632 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44856 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:18:30 [loggers.py:273] Engine 000: Avg prompt throughput: 2351.1 tokens/s, Avg generation throughput: 55.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44480 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32874 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:18:40 [loggers.py:273] Engine 000: Avg prompt throughput: 500.7 tokens/s, Avg generation throughput: 241.9 tokens/s, Running: 60 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:18:50 [loggers.py:273] Engine 000: Avg prompt throughput: 234.3 tokens/s, Avg generation throughput: 613.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:19:00 [loggers.py:273] Engine 000: Avg prompt throughput: 80.0 tokens/s, Avg generation throughput: 669.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:19:10 [loggers.py:273] Engine 000: Avg prompt throughput: 97.0 tokens/s, Avg generation throughput: 656.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:19:20 [loggers.py:273] Engine 000: Avg prompt throughput: 52.8 tokens/s, Avg generation throughput: 669.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:19:30 [loggers.py:273] Engine 000: Avg prompt throughput: 37.2 tokens/s, Avg generation throughput: 677.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:19:40 [loggers.py:273] Engine 000: Avg prompt throughput: 22.9 tokens/s, Avg generation throughput: 684.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:19:50 [loggers.py:273] Engine 000: Avg prompt throughput: 33.6 tokens/s, Avg generation throughput: 677.4 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:20:00 [loggers.py:273] Engine 000: Avg prompt throughput: 46.3 tokens/s, Avg generation throughput: 670.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:20:10 [loggers.py:273] Engine 000: Avg prompt throughput: 26.3 tokens/s, Avg generation throughput: 677.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:20:20 [loggers.py:273] Engine 000: Avg prompt throughput: 37.4 tokens/s, Avg generation throughput: 677.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:20:30 [loggers.py:273] Engine 000: Avg prompt throughput: 11.3 tokens/s, Avg generation throughput: 684.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:20:40 [loggers.py:273] Engine 000: Avg prompt throughput: 199.7 tokens/s, Avg generation throughput: 664.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32874 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:20:50 [loggers.py:273] Engine 000: Avg prompt throughput: 227.6 tokens/s, Avg generation throughput: 599.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:21:00 [loggers.py:273] Engine 000: Avg prompt throughput: 185.2 tokens/s, Avg generation throughput: 627.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:21:10 [loggers.py:273] Engine 000: Avg prompt throughput: 158.2 tokens/s, Avg generation throughput: 634.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:21:20 [loggers.py:273] Engine 000: Avg prompt throughput: 67.4 tokens/s, Avg generation throughput: 662.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:21:30 [loggers.py:273] Engine 000: Avg prompt throughput: 147.3 tokens/s, Avg generation throughput: 635.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:21:40 [loggers.py:273] Engine 000: Avg prompt throughput: 40.7 tokens/s, Avg generation throughput: 676.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:21:50 [loggers.py:273] Engine 000: Avg prompt throughput: 43.6 tokens/s, Avg generation throughput: 683.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:22:00 [loggers.py:273] Engine 000: Avg prompt throughput: 61.4 tokens/s, Avg generation throughput: 669.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:22:10 [loggers.py:273] Engine 000: Avg prompt throughput: 5.5 tokens/s, Avg generation throughput: 684.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:22:20 [loggers.py:273] Engine 000: Avg prompt throughput: 25.5 tokens/s, Avg generation throughput: 684.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:22:30 [loggers.py:273] Engine 000: Avg prompt throughput: 37.9 tokens/s, Avg generation throughput: 683.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:22:40 [loggers.py:273] Engine 000: Avg prompt throughput: 14.4 tokens/s, Avg generation throughput: 684.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32874 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32874 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:22:50 [loggers.py:273] Engine 000: Avg prompt throughput: 136.4 tokens/s, Avg generation throughput: 647.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:23:00 [loggers.py:273] Engine 000: Avg prompt throughput: 145.8 tokens/s, Avg generation throughput: 648.4 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:23:10 [loggers.py:273] Engine 000: Avg prompt throughput: 128.2 tokens/s, Avg generation throughput: 654.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:23:20 [loggers.py:273] Engine 000: Avg prompt throughput: 91.2 tokens/s, Avg generation throughput: 662.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:23:30 [loggers.py:273] Engine 000: Avg prompt throughput: 70.1 tokens/s, Avg generation throughput: 662.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:23:40 [loggers.py:273] Engine 000: Avg prompt throughput: 112.3 tokens/s, Avg generation throughput: 655.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:23:50 [loggers.py:273] Engine 000: Avg prompt throughput: 44.8 tokens/s, Avg generation throughput: 683.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33070 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33130 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32776 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33074 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:24:00 [loggers.py:273] Engine 000: Avg prompt throughput: 50.8 tokens/s, Avg generation throughput: 658.1 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60964 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60926 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32874 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33098 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33090 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:24:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 578.6 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33028 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60908 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:24:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 541.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:60858 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32956 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:24:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 520.8 tokens/s, Running: 46 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33036 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:24:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 503.6 tokens/s, Running: 45 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32840 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60892 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32940 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60960 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33024 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:24:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 450.3 tokens/s, Running: 36 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32790 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32988 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32972 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33060 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:25:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 361.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32902 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32854 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32980 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33008 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60904 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33154 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33044 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32990 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:25:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 267.1 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:33106 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33114 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33042 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60976 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60884 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60998 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60868 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:25:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 173.6 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32780 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32778 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32770 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:25:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 92.2 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32920 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33142 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:32900 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60944 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:60912 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:33134 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:25:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 27.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:32826 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:25:50 [loggers.py:273] Engine 000: Avg prompt throughput: 1252.4 tokens/s, Avg generation throughput: 314.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:26:00 [loggers.py:273] Engine 000: Avg prompt throughput: 180.1 tokens/s, Avg generation throughput: 654.9 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:26:10 [loggers.py:273] Engine 000: Avg prompt throughput: 246.3 tokens/s, Avg generation throughput: 634.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:26:20 [loggers.py:273] Engine 000: Avg prompt throughput: 257.5 tokens/s, Avg generation throughput: 640.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:26:30 [loggers.py:273] Engine 000: Avg prompt throughput: 281.9 tokens/s, Avg generation throughput: 613.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:26:40 [loggers.py:273] Engine 000: Avg prompt throughput: 241.9 tokens/s, Avg generation throughput: 634.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:26:50 [loggers.py:273] Engine 000: Avg prompt throughput: 197.6 tokens/s, Avg generation throughput: 648.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:27:00 [loggers.py:273] Engine 000: Avg prompt throughput: 137.7 tokens/s, Avg generation throughput: 662.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:27:10 [loggers.py:273] Engine 000: Avg prompt throughput: 174.6 tokens/s, Avg generation throughput: 648.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:27:20 [loggers.py:273] Engine 000: Avg prompt throughput: 84.3 tokens/s, Avg generation throughput: 676.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:27:30 [loggers.py:273] Engine 000: Avg prompt throughput: 94.9 tokens/s, Avg generation throughput: 669.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:27:40 [loggers.py:273] Engine 000: Avg prompt throughput: 149.0 tokens/s, Avg generation throughput: 655.5 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:27:50 [loggers.py:273] Engine 000: Avg prompt throughput: 44.3 tokens/s, Avg generation throughput: 682.4 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:28:00 [loggers.py:273] Engine 000: Avg prompt throughput: 667.1 tokens/s, Avg generation throughput: 574.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:28:10 [loggers.py:273] Engine 000: Avg prompt throughput: 274.4 tokens/s, Avg generation throughput: 626.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:28:20 [loggers.py:273] Engine 000: Avg prompt throughput: 284.4 tokens/s, Avg generation throughput: 620.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:28:30 [loggers.py:273] Engine 000: Avg prompt throughput: 226.2 tokens/s, Avg generation throughput: 627.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:28:40 [loggers.py:273] Engine 000: Avg prompt throughput: 195.4 tokens/s, Avg generation throughput: 655.0 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:28:50 [loggers.py:273] Engine 000: Avg prompt throughput: 330.4 tokens/s, Avg generation throughput: 612.5 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:29:00 [loggers.py:273] Engine 000: Avg prompt throughput: 250.2 tokens/s, Avg generation throughput: 628.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:29:10 [loggers.py:273] Engine 000: Avg prompt throughput: 162.7 tokens/s, Avg generation throughput: 648.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:29:20 [loggers.py:273] Engine 000: Avg prompt throughput: 331.6 tokens/s, Avg generation throughput: 600.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:29:30 [loggers.py:273] Engine 000: Avg prompt throughput: 182.6 tokens/s, Avg generation throughput: 642.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:29:40 [loggers.py:273] Engine 000: Avg prompt throughput: 237.2 tokens/s, Avg generation throughput: 634.0 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:29:50 [loggers.py:273] Engine 000: Avg prompt throughput: 241.3 tokens/s, Avg generation throughput: 628.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:30:00 [loggers.py:273] Engine 000: Avg prompt throughput: 185.7 tokens/s, Avg generation throughput: 648.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:30:10 [loggers.py:273] Engine 000: Avg prompt throughput: 167.0 tokens/s, Avg generation throughput: 655.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:30:20 [loggers.py:273] Engine 000: Avg prompt throughput: 128.0 tokens/s, Avg generation throughput: 649.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:30:30 [loggers.py:273] Engine 000: Avg prompt throughput: 269.0 tokens/s, Avg generation throughput: 620.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:30:40 [loggers.py:273] Engine 000: Avg prompt throughput: 246.5 tokens/s, Avg generation throughput: 627.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:30:50 [loggers.py:273] Engine 000: Avg prompt throughput: 204.9 tokens/s, Avg generation throughput: 655.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:31:00 [loggers.py:273] Engine 000: Avg prompt throughput: 294.8 tokens/s, Avg generation throughput: 620.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:31:10 [loggers.py:273] Engine 000: Avg prompt throughput: 187.8 tokens/s, Avg generation throughput: 648.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:31:20 [loggers.py:273] Engine 000: Avg prompt throughput: 352.8 tokens/s, Avg generation throughput: 613.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:31:30 [loggers.py:273] Engine 000: Avg prompt throughput: 126.7 tokens/s, Avg generation throughput: 662.6 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:31:40 [loggers.py:273] Engine 000: Avg prompt throughput: 138.3 tokens/s, Avg generation throughput: 649.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:31:50 [loggers.py:273] Engine 000: Avg prompt throughput: 254.4 tokens/s, Avg generation throughput: 627.8 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:32:00 [loggers.py:273] Engine 000: Avg prompt throughput: 234.5 tokens/s, Avg generation throughput: 627.7 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:32:10 [loggers.py:273] Engine 000: Avg prompt throughput: 240.3 tokens/s, Avg generation throughput: 620.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:32:20 [loggers.py:273] Engine 000: Avg prompt throughput: 288.7 tokens/s, Avg generation throughput: 620.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:32:30 [loggers.py:273] Engine 000: Avg prompt throughput: 192.5 tokens/s, Avg generation throughput: 642.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:32:40 [loggers.py:273] Engine 000: Avg prompt throughput: 372.0 tokens/s, Avg generation throughput: 605.8 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:32:50 [loggers.py:273] Engine 000: Avg prompt throughput: 256.7 tokens/s, Avg generation throughput: 627.7 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:33:00 [loggers.py:273] Engine 000: Avg prompt throughput: 246.5 tokens/s, Avg generation throughput: 628.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44748 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44818 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45254 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44966 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45200 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44950 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:33:10 [loggers.py:273] Engine 000: Avg prompt throughput: 91.5 tokens/s, Avg generation throughput: 636.3 tokens/s, Running: 57 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44774 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44916 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45006 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44896 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44860 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44894 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45034 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:33:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 592.6 tokens/s, Running: 50 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45196 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45084 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45136 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45194 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45166 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45188 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45056 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44782 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44882 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45266 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:33:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 488.7 tokens/s, Running: 39 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44984 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:33:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 418.9 tokens/s, Running: 38 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45210 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44910 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44804 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44968 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45240 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44938 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:33:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 380.4 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44930 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45182 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44802 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45100 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:34:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 318.2 tokens/s, Running: 27 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45018 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45238 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45148 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:34:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 269.3 tokens/s, Running: 24 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45268 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44996 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45180 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45108 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44848 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:34:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 235.5 tokens/s, Running: 19 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45174 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45202 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:34:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 204.8 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.5%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44796 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44876 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45228 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45168 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44760 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45132 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:34:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 156.2 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44864 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44958 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44932 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44834 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45152 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:44994 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:34:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 72.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:45120 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45220 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:45178 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:35:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 30.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:44784 - "POST /v1/chat/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:35:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:35:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:41828 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1142782) INFO: 127.0.0.1:41828 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1142782) INFO: 127.0.0.1:41828 - "GET /tokenizer_info HTTP/1.1" 404 Not Found (APIServer pid=1142782) INFO: 127.0.0.1:36472 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36312 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:36:00 [loggers.py:273] Engine 000: Avg prompt throughput: 1335.3 tokens/s, Avg generation throughput: 466.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36506 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:36:10 [loggers.py:273] Engine 000: Avg prompt throughput: 12.6 tokens/s, Avg generation throughput: 697.3 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:36:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 703.9 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:36:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:36:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.6%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:36:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:37:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:37:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 697.5 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36364 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:37:20 [loggers.py:273] Engine 000: Avg prompt throughput: 15.3 tokens/s, Avg generation throughput: 697.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36198 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36200 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36206 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36222 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36226 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36252 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36272 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36280 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36292 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36308 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36328 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36336 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36340 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36372 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36402 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36412 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36428 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36430 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36452 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36456 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36464 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36528 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36534 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36548 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36560 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36602 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36618 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36628 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36646 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36662 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36664 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36676 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36690 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36704 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36712 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36726 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36728 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36472 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:37:30 [loggers.py:273] Engine 000: Avg prompt throughput: 694.7 tokens/s, Avg generation throughput: 634.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36312 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36456 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36506 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:37:40 [loggers.py:273] Engine 000: Avg prompt throughput: 24.6 tokens/s, Avg generation throughput: 684.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:37:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:38:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:38:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:38:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:38:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.2 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:38:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:38:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 691.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36364 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36412 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:39:00 [loggers.py:273] Engine 000: Avg prompt throughput: 15.6 tokens/s, Avg generation throughput: 684.4 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.9%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36206 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36226 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36200 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36198 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36222 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36272 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36252 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36328 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36336 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36308 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36280 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36340 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36292 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36430 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36402 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36372 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36428 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36452 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36528 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36548 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36534 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36464 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36628 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36646 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36664 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36602 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36662 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36676 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36618 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36690 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36560 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36704 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36726 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36728 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36712 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36472 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36312 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36456 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:39:10 [loggers.py:273] Engine 000: Avg prompt throughput: 228.5 tokens/s, Avg generation throughput: 458.1 tokens/s, Running: 33 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36506 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:39:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 352.5 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:39:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 348.8 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:39:40 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 352.0 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.2%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:39:50 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 345.6 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:40:00 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 348.8 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:40:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 348.7 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:40:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 345.6 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36364 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36412 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:40:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 340.1 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.3%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:36184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36206 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36226 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36200 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36190 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36198 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36222 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36232 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36272 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36252 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36328 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36336 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36308 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36280 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36340 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36292 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36430 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36402 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:36372 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:40:40 [loggers.py:273] Engine 000: Avg prompt throughput: 1820.1 tokens/s, Avg generation throughput: 247.2 tokens/s, Running: 19 reqs, Waiting: 45 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58386 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58320 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:40:50 [loggers.py:273] Engine 000: Avg prompt throughput: 5880.3 tokens/s, Avg generation throughput: 223.0 tokens/s, Running: 60 reqs, Waiting: 1 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58344 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58584 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58436 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58344 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58386 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:41:00 [loggers.py:273] Engine 000: Avg prompt throughput: 5250.9 tokens/s, Avg generation throughput: 286.0 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58320 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58344 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58436 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:41:10 [loggers.py:273] Engine 000: Avg prompt throughput: 4763.9 tokens/s, Avg generation throughput: 324.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58386 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58320 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58436 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58386 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:41:20 [loggers.py:273] Engine 000: Avg prompt throughput: 4476.3 tokens/s, Avg generation throughput: 344.6 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58386 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58320 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58436 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:41:30 [loggers.py:273] Engine 000: Avg prompt throughput: 4481.6 tokens/s, Avg generation throughput: 356.3 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58320 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58584 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:41:40 [loggers.py:273] Engine 000: Avg prompt throughput: 4289.5 tokens/s, Avg generation throughput: 384.2 tokens/s, Running: 63 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.4%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58344 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58584 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:41:50 [loggers.py:273] Engine 000: Avg prompt throughput: 3783.6 tokens/s, Avg generation throughput: 391.6 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.7%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58332 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58122 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58240 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58302 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58376 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58538 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58250 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58584 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58522 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58406 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58128 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58420 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58610 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58314 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58542 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58202 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58422 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58248 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58634 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58440 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58586 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58550 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58230 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58574 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58632 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58100 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58358 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58098 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58266 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58160 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58216 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58514 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58344 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58184 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58282 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58418 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58454 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58400 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58106 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58276 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58600 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58554 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58388 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58568 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58386 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58616 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:42:00 [loggers.py:273] Engine 000: Avg prompt throughput: 842.5 tokens/s, Avg generation throughput: 348.1 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58436 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58320 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58208 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58546 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58304 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58296 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58438 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58144 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:42:10 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 164.9 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.8%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO: 127.0.0.1:58500 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58362 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58474 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58496 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58186 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58518 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58468 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58482 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58354 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO: 127.0.0.1:58168 - "POST /v1/completions HTTP/1.1" 200 OK (APIServer pid=1142782) INFO 07-24 00:42:20 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:42:30 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% (APIServer pid=1142782) INFO 07-24 00:45:00 [launcher.py:100] [shutdown] API server: shutdown triggered (APIServer pid=1142782) INFO 07-24 00:45:00 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s (APIServer pid=1142782) INFO 07-24 00:45:00 [core_client.py:655] [shutdown] MPClient: start timeout=0s (APIServer pid=1142782) INFO 07-24 00:45:00 [core_client.py:657] [shutdown] MPClient: stopping engine manager (APIServer pid=1142782) WARNING 07-24 00:45:00 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1 (EngineCore pid=1142928) INFO 07-24 00:45:00 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM (EngineCore pid=1142928) INFO 07-24 00:45:00 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s (EngineCore pid=1142928) INFO 07-24 00:45:00 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown (EngineCore pid=1142928) INFO 07-24 00:45:00 [core.py:1227] [shutdown] EngineCore: exiting busy loop (Worker_PP0 pid=1143042) INFO 07-24 00:45:00 [multiproc_executor.py:790] Parent process exited, terminating worker queues (EngineCore pid=1142928) INFO 07-24 00:45:00 [multiproc_executor.py:426] [shutdown] Executor: waiting for worker exit count=2 (APIServer pid=1142782) INFO 07-24 00:45:00 [core_client.py:659] [shutdown] MPClient: engine manager stopped (APIServer pid=1142782) INFO 07-24 00:45:00 [core_client.py:660] [shutdown] MPClient: cleaning up background resources (APIServer pid=1142782) INFO 07-24 00:45:00 [core_client.py:662] [shutdown] MPClient: complete (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] AsyncLLM output_handler failed. (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] Traceback (most recent call last): (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] outputs = await engine_core.get_output_async() (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] File "/home/henry/Documents/PythonProjects/variable-reap/vllm-plugin/.venv25/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1061, in get_output_async (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] raise self._format_exception(outputs) from None (APIServer pid=1142782) ERROR 07-24 00:45:00 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (APIServer pid=1142782) INFO 07-24 00:45:00 [launcher.py:125] [shutdown] API server: engine client stopped (APIServer pid=1142782) INFO 07-24 00:45:00 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown (APIServer pid=1142782) INFO 07-24 00:45:00 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server (APIServer pid=1142782) INFO: Shutting down (APIServer pid=1142782) INFO: Waiting for application shutdown. (APIServer pid=1142782) INFO: Application shutdown complete. /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 3 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '