Hugging Face's logo

hbfreed
/
variable-reap-archive

Safetensors
Model card Files
xet
 Community
variable-reap-archive / evals /release /math-keep25-step0150 /server.log
hbfreed's picture
hbfreed
Add files using upload-large-folder tool
a82fb9e verified
Raw
History Blame Contribute Delete
309 kB
(APIServer pid=77325) INFO 08-08 22:01:45 [api_utils.py:339]
(APIServer pid=77325) INFO 08-08 22:01:45 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=77325) INFO 08-08 22:01:45 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0
(APIServer pid=77325) INFO 08-08 22:01:45 [api_utils.py:339] █▄█▀ █ █ █ █ model outputs/release/healed/release-offpolicy-math-keep25/step0150
(APIServer pid=77325) INFO 08-08 22:01:45 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=77325) INFO 08-08 22:01:45 [api_utils.py:339]
(APIServer pid=77325) INFO 08-08 22:01:45 [api_utils.py:273] non-default args: {'model_tag': 'outputs/release/healed/release-offpolicy-math-keep25/step0150', 'host': '127.0.0.1', 'port': 8460, 'model': 'outputs/release/healed/release-offpolicy-math-keep25/step0150', 'max_model_len': 4096, 'enforce_eager': True, 'served_model_name': ['student'], 'gpu_memory_utilization': 0.85}
(APIServer pid=77325) INFO 08-08 22:01:45 [model.py:619] Resolved architecture: PrunedOlmoeForCausalLM
(APIServer pid=77325) INFO 08-08 22:01:45 [model.py:1776] Using max model len 4096
(APIServer pid=77325) INFO 08-08 22:01:45 [vllm.py:1042] Asynchronous scheduling is enabled.
(APIServer pid=77325) WARNING 08-08 22:01:45 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(APIServer pid=77325) WARNING 08-08 22:01:45 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(APIServer pid=77325) INFO 08-08 22:01:45 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(APIServer pid=77325) INFO 08-08 22:01:46 [vllm.py:1322] Cudagraph is disabled under eager mode
(APIServer pid=77325) INFO 08-08 22:01:46 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=77507) INFO 08-08 22:01:53 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='outputs/release/healed/release-offpolicy-math-keep25/step0150', speculative_config=None, tokenizer='outputs/release/healed/release-offpolicy-math-keep25/step0150', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=student, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'ir_enable_torch_wrap': False, 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=77507) INFO 08-08 22:01:54 [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.0.27:34643 backend=nccl
(EngineCore pid=77507) INFO 08-08 22:01:54 [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=77507) INFO 08-08 22:01:55 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=77507) INFO 08-08 22:01:55 [gpu_model_runner.py:5209] Starting to load model outputs/release/healed/release-offpolicy-math-keep25/step0150...
(EngineCore pid=77507) INFO 08-08 22:01:55 [cuda.py:476] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=77507) INFO 08-08 22:01:55 [flash_attn.py:718] Using FlashAttention version 2
(EngineCore pid=77507) INFO 08-08 22:01:56 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 3.89 GiB. Available RAM: 109.33 GiB.
(EngineCore pid=77507) INFO 08-08 22:01:56 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=77507) Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore pid=77507) Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:17<00:00, 17.82s/it]
(EngineCore pid=77507) Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:17<00:00, 17.82s/it]
(EngineCore pid=77507)
(EngineCore pid=77507) INFO 08-08 22:02:13 [default_loader.py:430] Loading weights took 17.84 seconds
(EngineCore pid=77507) INFO 08-08 22:02:14 [gpu_model_runner.py:5306] Model loading took 3.89 GiB memory and 18.063609 seconds
(EngineCore pid=77507) INFO 08-08 22:02:16 [gpu_worker.py:538] Available KV cache memory: 15.86 GiB
(EngineCore pid=77507) INFO 08-08 22:02:16 [kv_cache_utils.py:2146] GPU KV cache size: 129,920 tokens
(EngineCore pid=77507) INFO 08-08 22:02:16 [kv_cache_utils.py:2147] Maximum concurrency for 4,096 tokens per request: 31.72x
(EngineCore pid=77507) INFO 08-08 22:02:17 [cutedsl_warmup.py:97] Skipping CuTeDSL warmup because no compile units were requested.
(EngineCore pid=77507) INFO 08-08 22:02:17 [jit_monitor.py:73] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=77507) INFO 08-08 22:02:17 [core.py:344] init engine (profile, create kv cache, warmup model) took 2.95 s
(EngineCore pid=77507) INFO 08-08 22:02:17 [vllm.py:1042] Asynchronous scheduling is enabled.
(EngineCore pid=77507) WARNING 08-08 22:02:17 [vllm.py:1096] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(EngineCore pid=77507) WARNING 08-08 22:02:17 [vllm.py:1144] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(EngineCore pid=77507) INFO 08-08 22:02:17 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(EngineCore pid=77507) INFO 08-08 22:02:17 [vllm.py:1322] Cudagraph is disabled under eager mode
(EngineCore pid=77507) INFO 08-08 22:02:17 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(APIServer pid=77325) INFO 08-08 22:02:17 [api_server.py:612] Supported tasks: ['generate']
(APIServer pid=77325) INFO 08-08 22:02:17 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=77325) INFO 08-08 22:02:17 [api_server.py:616] Starting vLLM server on http://127.0.0.1:8460
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:37] Available routes are:
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=77325) INFO 08-08 22:02:17 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=77325) INFO: Started server process [77325]
(APIServer pid=77325) INFO: Waiting for application startup.
(APIServer pid=77325) INFO: Application startup complete.
(APIServer pid=77325) INFO: 127.0.0.1:45286 - "GET /health HTTP/1.1" 200 OK
(EngineCore pid=77507) WARNING 08-08 22:02:39 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _plan_tiles. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=77507) WARNING 08-08 22:02:39 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _gate_up_silu. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=77507) WARNING 08-08 22:02:39 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _down_proj. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=77507) WARNING 08-08 22:02:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _count_experts. This causes a latency spike; consider extending warmup to cover this shape/config.
(EngineCore pid=77507) WARNING 08-08 22:02:42 [jit_monitor.py:129] Triton kernel JIT compilation during inference: _scatter_routes. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:02:48 [loggers.py:273] Engine 000: Avg prompt throughput: 753.7 tokens/s, Avg generation throughput: 2455.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.1%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:02:58 [loggers.py:273] Engine 000: Avg prompt throughput: 704.4 tokens/s, Avg generation throughput: 3271.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.2%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:03:08 [loggers.py:273] Engine 000: Avg prompt throughput: 700.2 tokens/s, Avg generation throughput: 3284.5 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.9%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:03:18 [loggers.py:273] Engine 000: Avg prompt throughput: 681.3 tokens/s, Avg generation throughput: 3289.8 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.6%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:03:28 [loggers.py:273] Engine 000: Avg prompt throughput: 629.1 tokens/s, Avg generation throughput: 3191.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.0%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:03:38 [loggers.py:273] Engine 000: Avg prompt throughput: 671.8 tokens/s, Avg generation throughput: 3218.2 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.3%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:03:48 [loggers.py:273] Engine 000: Avg prompt throughput: 640.6 tokens/s, Avg generation throughput: 3218.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:03:58 [loggers.py:273] Engine 000: Avg prompt throughput: 645.3 tokens/s, Avg generation throughput: 3228.7 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:04:08 [loggers.py:273] Engine 000: Avg prompt throughput: 758.9 tokens/s, Avg generation throughput: 3316.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.9%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:04:18 [loggers.py:273] Engine 000: Avg prompt throughput: 732.7 tokens/s, Avg generation throughput: 3313.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.8%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:04:28 [loggers.py:273] Engine 000: Avg prompt throughput: 682.8 tokens/s, Avg generation throughput: 3198.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.2%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:04:38 [loggers.py:273] Engine 000: Avg prompt throughput: 671.0 tokens/s, Avg generation throughput: 3186.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:04:48 [loggers.py:273] Engine 000: Avg prompt throughput: 619.7 tokens/s, Avg generation throughput: 3225.0 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.5%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:04:58 [loggers.py:273] Engine 000: Avg prompt throughput: 628.8 tokens/s, Avg generation throughput: 3338.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.8%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:05:08 [loggers.py:273] Engine 000: Avg prompt throughput: 665.7 tokens/s, Avg generation throughput: 3305.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.2%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48722 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48906 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48694 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:05:18 [loggers.py:273] Engine 000: Avg prompt throughput: 661.8 tokens/s, Avg generation throughput: 3335.0 tokens/s, Running: 42 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.8%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48748 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48846 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:49000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48634 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48642 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48832 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48710 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48886 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48860 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48792 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48686 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:05:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 878.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 0.0%
(APIServer pid=77325) INFO: 127.0.0.1:48780 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:48892 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:05:38 [loggers.py:273] Engine 000: Avg prompt throughput: 538.2 tokens/s, Avg generation throughput: 927.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.7%, Prefix cache hit rate: 29.9%
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:05:48 [loggers.py:273] Engine 000: Avg prompt throughput: 510.7 tokens/s, Avg generation throughput: 2848.5 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.4%, Prefix cache hit rate: 47.3%
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:05:58 [loggers.py:273] Engine 000: Avg prompt throughput: 391.9 tokens/s, Avg generation throughput: 2630.6 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.7%, Prefix cache hit rate: 55.7%
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:06:08 [loggers.py:273] Engine 000: Avg prompt throughput: 520.9 tokens/s, Avg generation throughput: 2590.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.0%, Prefix cache hit rate: 62.3%
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:06:18 [loggers.py:273] Engine 000: Avg prompt throughput: 384.4 tokens/s, Avg generation throughput: 2680.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.4%, Prefix cache hit rate: 66.0%
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:06:28 [loggers.py:273] Engine 000: Avg prompt throughput: 539.1 tokens/s, Avg generation throughput: 2543.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.7%, Prefix cache hit rate: 69.6%
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:06:38 [loggers.py:273] Engine 000: Avg prompt throughput: 386.9 tokens/s, Avg generation throughput: 2686.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 73.0%
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:06:48 [loggers.py:273] Engine 000: Avg prompt throughput: 455.2 tokens/s, Avg generation throughput: 2611.3 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.1%, Prefix cache hit rate: 75.1%
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:06:58 [loggers.py:273] Engine 000: Avg prompt throughput: 375.1 tokens/s, Avg generation throughput: 2624.9 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.9%, Prefix cache hit rate: 77.4%
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37302 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:07:08 [loggers.py:273] Engine 000: Avg prompt throughput: 426.4 tokens/s, Avg generation throughput: 2658.2 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.6%, Prefix cache hit rate: 79.1%
(APIServer pid=77325) INFO: 127.0.0.1:37368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37540 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37310 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37504 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37574 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37408 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37582 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37470 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37616 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37350 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37612 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:07:18 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1761.9 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.5%, Prefix cache hit rate: 79.1%
(APIServer pid=77325) INFO: 127.0.0.1:37492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37334 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37384 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:37660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:07:28 [loggers.py:273] Engine 000: Avg prompt throughput: 318.3 tokens/s, Avg generation throughput: 1324.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.1%, Prefix cache hit rate: 79.3%
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:07:38 [loggers.py:273] Engine 000: Avg prompt throughput: 262.7 tokens/s, Avg generation throughput: 3313.2 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.8%, Prefix cache hit rate: 79.4%
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:07:48 [loggers.py:273] Engine 000: Avg prompt throughput: 271.3 tokens/s, Avg generation throughput: 2969.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.4%, Prefix cache hit rate: 79.7%
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:07:58 [loggers.py:273] Engine 000: Avg prompt throughput: 302.0 tokens/s, Avg generation throughput: 2946.8 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.5%, Prefix cache hit rate: 79.9%
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:08:08 [loggers.py:273] Engine 000: Avg prompt throughput: 238.3 tokens/s, Avg generation throughput: 2646.0 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 80.1%
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:08:18 [loggers.py:273] Engine 000: Avg prompt throughput: 216.8 tokens/s, Avg generation throughput: 2680.1 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 80.4%
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:08:28 [loggers.py:273] Engine 000: Avg prompt throughput: 323.8 tokens/s, Avg generation throughput: 2763.7 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.8%, Prefix cache hit rate: 80.5%
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:08:38 [loggers.py:273] Engine 000: Avg prompt throughput: 294.9 tokens/s, Avg generation throughput: 2784.0 tokens/s, Running: 47 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.7%, Prefix cache hit rate: 80.7%
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:08:48 [loggers.py:273] Engine 000: Avg prompt throughput: 195.5 tokens/s, Avg generation throughput: 2604.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.8%, Prefix cache hit rate: 80.9%
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:08:58 [loggers.py:273] Engine 000: Avg prompt throughput: 312.3 tokens/s, Avg generation throughput: 2692.4 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.7%, Prefix cache hit rate: 81.2%
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:09:08 [loggers.py:273] Engine 000: Avg prompt throughput: 303.0 tokens/s, Avg generation throughput: 2840.5 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.3%, Prefix cache hit rate: 81.3%
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58108 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58282 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58520 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58452 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58464 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58216 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58372 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58548 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58318 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58534 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:09:18 [loggers.py:273] Engine 000: Avg prompt throughput: 195.7 tokens/s, Avg generation throughput: 2702.9 tokens/s, Running: 29 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.2%, Prefix cache hit rate: 80.4%
(APIServer pid=77325) INFO: 127.0.0.1:58312 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58392 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58494 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58118 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58336 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58400 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58138 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58558 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58172 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58152 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:09:28 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1159.3 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 5.7%, Prefix cache hit rate: 80.4%
(APIServer pid=77325) INFO: 127.0.0.1:58562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58188 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:58112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:09:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 65.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.4%
(APIServer pid=77325) INFO 08-08 22:09:48 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 80.4%
(APIServer pid=77325) INFO: 127.0.0.1:57550 - "GET /tokenizer_info HTTP/1.1" 404 Not Found
(APIServer pid=77325) INFO: 127.0.0.1:57550 - "GET /tokenizer_info HTTP/1.1" 404 Not Found
(APIServer pid=77325) INFO: 127.0.0.1:57550 - "GET /tokenizer_info HTTP/1.1" 404 Not Found
(APIServer pid=77325) INFO: 127.0.0.1:41316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41388 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41574 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41426 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41454 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41388 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41388 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41426 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41454 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41586 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41638 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41280 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41396 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41360 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41514 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41348 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41504 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41698 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41414 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41488 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41316 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41654 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41524 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:10:28 [loggers.py:273] Engine 000: Avg prompt throughput: 2324.7 tokens/s, Avg generation throughput: 2280.2 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.1%, Prefix cache hit rate: 70.3%
(APIServer pid=77325) INFO: 127.0.0.1:41426 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41454 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41388 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41372 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41382 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41472 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41606 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41522 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41284 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41292 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41298 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41338 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41434 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41450 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41526 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41558 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41594 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41742 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41542 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41574 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41406 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41466 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41394 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:10:38 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1381.1 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.7%, Prefix cache hit rate: 70.3%
(APIServer pid=77325) INFO: 127.0.0.1:41640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:41478 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44652 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44652 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:10:48 [loggers.py:273] Engine 000: Avg prompt throughput: 4095.3 tokens/s, Avg generation throughput: 2371.2 tokens/s, Running: 48 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.3%, Prefix cache hit rate: 62.3%
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44652 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44652 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44652 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44652 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44704 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44670 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44914 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44880 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44814 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44766 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44900 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44716 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44706 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44664 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44778 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44884 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44662 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44806 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44870 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44576 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44640 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44682 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44866 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44768 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44602 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44782 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44584 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44590 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44572 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44650 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44810 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44578 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44714 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44678 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44588 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44622 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44628 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44812 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44690 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44868 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44652 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44570 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44792 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44754 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44830 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44624 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44922 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44746 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44846 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44852 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44612 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO: 127.0.0.1:44732 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=77325) INFO 08-08 22:10:58 [loggers.py:273] Engine 000: Avg prompt throughput: 1452.8 tokens/s, Avg generation throughput: 2274.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3%
(APIServer pid=77325) INFO 08-08 22:11:08 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 73.3%
(APIServer pid=77325) INFO 08-08 22:15:10 [launcher.py:100] [shutdown] API server: shutdown triggered
(APIServer pid=77325) INFO 08-08 22:15:10 [launcher.py:116] [shutdown] API server: stopping engine client mode=abort timeout=0s
(APIServer pid=77325) INFO 08-08 22:15:10 [core_client.py:655] [shutdown] MPClient: start timeout=0s
(APIServer pid=77325) INFO 08-08 22:15:10 [core_client.py:657] [shutdown] MPClient: stopping engine manager
(APIServer pid=77325) WARNING 08-08 22:15:10 [utils.py:626] [shutdown] Process manager: force killing remaining processes count=1
(EngineCore pid=77507) INFO 08-08 22:15:10 [core.py:1214] [shutdown] EngineCore: trigger received signal=SIGTERM
(EngineCore pid=77507) INFO 08-08 22:15:10 [core.py:1333] [shutdown] EngineCore: start mode=abort timeout=0s
(EngineCore pid=77507) INFO 08-08 22:15:10 [core.py:1364] [shutdown] EngineCore: request processing complete; starting resource teardown
(EngineCore pid=77507) INFO 08-08 22:15:10 [core.py:1227] [shutdown] EngineCore: exiting busy loop
(APIServer pid=77325) INFO 08-08 22:15:10 [core_client.py:659] [shutdown] MPClient: engine manager stopped
(APIServer pid=77325) INFO 08-08 22:15:10 [core_client.py:660] [shutdown] MPClient: cleaning up background resources
(APIServer pid=77325) INFO 08-08 22:15:10 [core_client.py:662] [shutdown] MPClient: complete
(APIServer pid=77325) INFO 08-08 22:15:10 [launcher.py:125] [shutdown] API server: engine client stopped
(APIServer pid=77325) INFO 08-08 22:15:10 [launcher.py:128] [shutdown] API server: signalling HTTP server shutdown
(APIServer pid=77325) INFO 08-08 22:15:10 [launcher.py:149] [shutdown] API server: shutting down FastAPI HTTP server
(APIServer pid=77325) INFO: Shutting down
(APIServer pid=77325) INFO: Waiting for application shutdown.
(APIServer pid=77325) INFO: Application shutdown complete.
/home/henry/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '